Analyze - Documentation and filtering

Refine the code that filters extra files, based on further testing with player builds.

Update the documentation to explain more about when Analyse can be used.

Add a non-recursive option to reduce the noise if you only want to analyze a player build (e.g. ignore the StreamingAssets contents)

Author: SkowronskiAndrew

Analyzer/AnalyzerTool.cs

@@ -1,4 +1,5 @@
 using System;
+using System.Collections.Generic;
 using System.Diagnostics;
 using System.IO;
 using UnityDataTools.Analyzer.SQLite;
@@ -10,7 +11,13 @@ public class AnalyzerTool
 {
     bool m_Verbose = false;
 
-    public int Analyze(string path, string databaseName, string searchPattern, bool skipReferences, bool verbose)
+    public int Analyze(
+        string path,
+        string databaseName,
+        string searchPattern,
+        bool skipReferences,
+        bool verbose,
+        bool noRecursion)
     {
         m_Verbose = verbose;
 
@@ -29,7 +36,11 @@ public int Analyze(string path, string databaseName, string searchPattern, bool
         var timer = new Stopwatch();
         timer.Start();
 
-        var files = Directory.GetFiles(path, searchPattern, SearchOption.AllDirectories);
+        var files = Directory.GetFiles(
+            path,
+            searchPattern,
+            noRecursion ? SearchOption.TopDirectoryOnly : SearchOption.AllDirectories);
+
         int i = 1;
         foreach (var file in files)
         {
@@ -65,30 +76,31 @@ public int Analyze(string path, string databaseName, string searchPattern, bool
     bool ShouldIgnoreFile(string file)
     {
         // Unfortunately there is no standard extension for AssetBundles, and SerializedFiles often have no extension at all.
-        // There is also no distinctive signature at the start of a SerializedFile to immediately recognize it
-        // (Unity Archives do have this).
-        // So to reduce noise in UnityDataTool output we filter out files that we have a high confidence are NOT SerializedFiles or Unity Archives.
+        // Also there is also no distinctive signature at the start of a SerializedFile to immediately recognize it based on its first bytes.
+        // This makes it difficult to use the "--search-pattern" argument to only pick those files.
+
+        // Hence to reduce noise in UnityDataTool output we filter out files that we have a high confidence are
+        // NOT SerializedFiles or Unity Archives.
 
         string fileName = Path.GetFileName(file);
         string extension = Path.GetExtension(file);
 
-        if ((fileName == ".DS_Store") || // Automatically ignore these annoying OS X style files meta files.
-            (fileName == "archive_dependencies.bin") ||
-            (fileName == "scene_info.bin") ||
-            (fileName == "app.info") ||
-            (extension == ".txt") ||
-            (extension == ".resS") ||
-            (extension == ".resource") ||
-            (extension == ".json") ||
-            (extension == ".dll") ||
-            (extension == ".pdb") ||
-            (extension == ".manifest") ||
-            (extension == ".entities") ||
-            (extension == ".entityheader"))
-            return true;
-        return false;
+        return IgnoredFileNames.Contains(fileName) || IgnoredExtensions.Contains(extension);
     }
 
+    // These lists are based on expected output files in Player, AssetBundle, Addressables and ECS builds.
+    // However this is by no means exhaustive.
+    private static readonly HashSet<string> IgnoredFileNames = new()
+    {
+        ".DS_Store", "boot.config", "archive_dependencies.bin", "scene_info.bin", "app.info", "link.xml",
+        "catalog.bin", "catalog.hash"
+    };
+
+    private static readonly HashSet<string> IgnoredExtensions = new()
+    {
+        ".txt", ".resS", ".resource", ".json", ".dll", ".pdb", ".exe", ".manifest", ".entities", ".entityheader"
+    };
+
     void ProcessFile(string file, string rootDirectory, SQLiteWriter writer, int fileIndex, int cntFiles)
     {
         try
@@ -148,7 +160,7 @@ void ProcessFile(string file, string rootDirectory, SQLiteWriter writer, int fil
         {
             EraseProgressLine();
             Console.Error.WriteLine();
-            //Console.Error.WriteLine($"File not supported: {file}"); // This is commented out because another codepath will output "failed to load"
+            //A "failed to load" error will already be logged by the UnityFileSystem library
         }
         catch (Exception e)
         {

Documentation/unity-content-format.md

@@ -50,6 +50,14 @@ To do so, the **ForceAlwaysWriteTypeTrees** Diagnostic Switch must be enabled in
 
 ![](./TypeTreeForPlayer.png)
 
+
+Note: The `Resources\unity default resources` file is shipped with the Unity Editor and is not rebuilt when doing a Player Build.  It does not have TypeTrees.  Hence it is normal that this file emits errors when analyzing a player build, even after rebuilding with TypeTrees enabled.  For example:
+
+```
+Error processing file: C:\TestProject\CompressedPlayer\TestProject_Data\Resources\unity default resources
+System.ArgumentException: Invalid object id.
+```
+
 For more information about TypeTrees see the following section.
 
 ## TypeTrees
@@ -61,7 +69,7 @@ definition exactly matches the Type definition used when the object was serializ
 Unity will attempt to match up the properties as best as it can, based on the property names and structure
 of the data.  This process is called a "Safe Binary Read" and is somewhat slower than the regular fast binary read path.
 
-TypeTrees are important in the case of AssetBundles, to avoid rebuilding and redistributing all AssetBundles after each minor upgrade of Unity or after doing minor changes to your MonoBehaviour and ScriptableObject serialization.  However there can be a noticable overhead to storing the TypeTrees in each AssetBundle, e.g. the header size of each SerializedFile is bigger.
+TypeTrees are important in the case of AssetBundles, to avoid rebuilding and redistributing all AssetBundles after each minor upgrade of Unity or after doing minor changes to your MonoBehaviour and ScriptableObject serialization.  However there can be a noticeable overhead to storing the TypeTrees in each AssetBundle, e.g. the header size of each SerializedFile is bigger.
 
 TypeTrees also make it possible to load an AssetBundle in the Editor, when testing game play.
 

UnityDataTool/Program.cs

@@ -24,6 +24,7 @@ public static async Task<int> Main(string[] args)
             var rOpt = new Option<bool>(aliases: new[] { "--extract-references", "-r" }) { IsHidden = true };
             var pOpt = new Option<string>(aliases: new[] { "--search-pattern", "-p" }, description: "File search pattern", getDefaultValue: () => "*");
             var vOpt = new Option<bool>(aliases: new[] { "--verbose", "-v" }, description: "Verbose output");
+            var recurseOpt = new Option<bool>(aliases: new[] { "--no-recurse" }, description: "Do not analyze contents of subdirectories inside path");
 
             var analyzeCommand = new Command("analyze", "Analyze AssetBundles or SerializedFiles.")
             {
@@ -32,13 +33,14 @@ public static async Task<int> Main(string[] args)
                 sOpt,
                 rOpt,
                 pOpt,
-                vOpt
+                vOpt,
+                recurseOpt
             };
 
             analyzeCommand.AddAlias("analyse");
             analyzeCommand.SetHandler(
-                (DirectoryInfo di, string o, bool s, bool r, string p, bool v) => Task.FromResult(HandleAnalyze(di, o, s, r, p, v)),
-                pathArg, oOpt, sOpt, rOpt, pOpt, vOpt);
+                (DirectoryInfo di, string o, bool s, bool r, string p, bool v, bool recurseOpt) => Task.FromResult(HandleAnalyze(di, o, s, r, p, v, recurseOpt)),
+                pathArg, oOpt, sOpt, rOpt, pOpt, vOpt, recurseOpt);
 
             rootCommand.AddCommand(analyzeCommand);
         }
@@ -132,7 +134,14 @@ enum DumpFormat
         Text,
     }
 
-    static int HandleAnalyze(DirectoryInfo path, string outputFile, bool skipReferences, bool extractReferences, string searchPattern, bool verbose)
+    static int HandleAnalyze(
+        DirectoryInfo path,
+        string outputFile,
+        bool skipReferences,
+        bool extractReferences,
+        string searchPattern,
+        bool verbose,
+        bool noRecurse)
     {
         var analyzer = new AnalyzerTool();
 
@@ -141,7 +150,7 @@ static int HandleAnalyze(DirectoryInfo path, string outputFile, bool skipReferen
             Console.WriteLine("WARNING: --extract-references, -r option is deprecated (references are now extracted by default)");
         }
 
-        return analyzer.Analyze(path.FullName, outputFile, searchPattern, skipReferences, verbose);
+        return analyzer.Analyze(path.FullName, outputFile, searchPattern, skipReferences, verbose, noRecurse);
     }
 
     static int HandleFindReferences(FileInfo databasePath, string outputFile, long? objectId, string objectName, string objectType, bool findAll)

UnityDataTool/README.md

@@ -33,48 +33,72 @@ The tool is invoked from the command line like this: `UnityDataTool [command] [c
 
 For a list of available commands run it like this: `UnityDataTool --help`
 
-For help on a specific command use `--help` along with the command name, for example: `UnityDataTool analyse --help`
+For help on a specific command use `--help` along with the command name, for example: `UnityDataTool analyze --help`
 
 
 # Commands
 
 ## analyze/analyse
 
-This command extracts information from AssetBundles and SerializedFiles and dumps the results
-into a SQLite database. 
+This command extracts information from Unity Archives (e.g. AssetBundles) and SerializedFiles and dumps the results into a SQLite database. 
 
-The command will fail if the SerializedFiles were built without TypeTrees, see [this topic](../Documentation/unity-content-format.md) for more information.
 The command takes the path of the folder containing the files to analyze as argument.
 
 It also provides the following options:
 * -o, --output-file \<database-filename\>: filename of the database that will be created, the
-  default is database.db.
+  default is database.db.  Any existing file by that name will be replaced by the data from running this command.
 * -s, --skip-references: skip CRC and reference (PPtrs) extraction. Faster processing and smaller
   database, but inaccurate duplicate asset detection and no references table.
-* -p, --search-pattern \<pattern\>: search pattern used to determine which files are AssetBundles.  The default is \*.  The * and ? characters are supported, but not regular expressions. The search is always recursive.
+* -p, --search-pattern \<pattern\>: search pattern used to determine which files are AssetBundles.  The default is \*.  The * and ? characters are supported. Regular expressions are not supported.
+* -v, --verbose: show more information during the analysis process, for example list any files that are ignored.
+* --no-recurse: do not recurse into sub-directories.
 
-Example: `UnityDataTool analyze /path/to/asset/bundles -o my_database.db -p *.bundle`
+Example: `UnityDataTool analyze /path/to/asset/bundles -o my_database.db -p "*.bundle"`
 
 **Refer to this [documentation](../Analyzer/README.md#How-to-use-the-database) for more information
 about the output database structure.**
 
-### Common Warnings during Analysis
+Note: If a SerializedFile is built without TypeTrees, then the command will not be able to extract information about the contained objects.  It will print an error similar to this example, then skip to the next file:
 
-The analysis search may find files that are not actually Archives or SerializedFiles, for example .manifest files, text dumps etc.
+```
+Error processing file: C:\Src\TestProject\Build\Player\TestProject_Data\level0
+System.ArgumentException: Invalid object id.
+```
+
+See [this topic](../Documentation/unity-content-format.md) for more information about TypeTrees.
+
+### Example Input to the Analyze command
+
+Example of directories that could be analyzed:
 
-This can lead to error messages like this: 
+* The output path of an AssetBundle build.
+* A folder inside the StreamingAssets folder of a Unity Player build. For example:
+  * The "StreamingAssets/aa" folder, containing AssetBundles from an Addressables build.
+  * The "StreamingAssets/ContentArchives" folder containing sub-scene content if your project uses [Entities](https://docs.unity3d.com/Packages/[email protected]/manual/content-management-intro.html).
+* The "Data" folder of a Unity Player build.
+  * By default, any AssetBundles or ContentArchives in the StreamingAssets folder would also be included. Use the "--no-recurse" option to avoid that.
+  * Compressed Player Builds are supported. The data.unity3d file will be analyzed the same way AssetBundles are.
+  * The structure and content of a Player varies based on the platform.  In some cases you may have to first extract the content out of a platform-specific container file prior to Analysis (for example .apk files on Android).
+
+### Filtering Other File Types
+
+Analyze works by trying to process all files in the provided path, assuming they are all Unity Archives or SerializedFiles.  Because there is no standard file extension for those files it is tricky to reliably distinguish these file types from the other files that may also be found in the build output.
+
+This can lead to error messages in the UnityDataTool output like this: 
 
 ```
-Failed to load 'C:\....\AssetBundles.manifest'. File may be corrupted or was serialized with a newer version of Unity.
+Failed to load 'C:\....\MyData.db'. File may be corrupted or was serialized with a newer version of Unity.
 ```
 
-In that case it is not a serious error, because the analyze process will continue and can still produce a perfectly valid database file.
+Typically these are not serious errors. The analyze process will continue, and can still produce a perfectly valid database file.  However if there are many messages like this it can obscure more important or unexpected failures.
+
+To reduce the number of these warnings, UnityDataTool automatically ignores common filenames and file paths that are found in Player, AssetBundle, Addressable or Entities builds.  For example ".txt, .json, .manifest".  When the `--verbose` option is passed each ignored file will be listed.
 
-If you use an extension of other naming convention for your AssetBundles, for example ".bundle", then you can avoid those warnings using the `-p .bundle` option to ignore .manifest and other files.  
+If you use an extension or other naming convention for your AssetBundles, for example ".bundle", then you can avoid those warnings using the `-p .bundle` option to ignore other files.
 
-For Player builds there is no single -p option that can catch all SerializedFiles (unless it is a compressed build generating a single data.unity3d file).
+For Player builds there is no single -p option that can catch all SerializedFiles (unless it is a compressed build with a single `data.unity3d` file).
 
-Overall it can be a good idea to avoid those errors, as noisy errors may hide more serious errors that would need your attention.
+The `--no-recurse` option can reduce the volume of these warnings.
 
 ## dump