Extra formats, 1:1 fixes and diff tool

Author: Matthew Lyons

.gitignore

@@ -7,6 +7,10 @@
 /spec/reports/
 /tmp/
 Data*/*
+Scripts/*
+
+ins1
+ins2
 
 # rspec failure tracking
 .rspec_status

.vscode/launch.json

@@ -9,7 +9,7 @@
             "type": "Ruby",
             "request": "launch",
             "program": "${workspaceRoot}/bin/rmxp_extractor",
-            "args": ["import"]
+            "args": ["export", "rb"]
         }
     ]
 }

Gemfile

@@ -7,5 +7,8 @@ gem "oj"
 gem "ruby-progressbar"
 gem "fileutils"
 gem "pathname"
-gem "json"
 gem "zlib"
+gem "toml-rb"
+gem "json"
+gem "yaml"
+gem "amazing_print"

Gemfile.lock

@@ -12,23 +12,31 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
+    amazing_print (1.4.0)
+    citrus (3.0.2)
     fileutils (1.6.0)
     json (2.6.1)
     oj (3.13.11)
     pathname (0.2.0)
     ruby-progressbar (1.11.0)
+    toml-rb (2.1.2)
+      citrus (~> 3.0, > 3.0)
+    yaml (0.1.1)
     zlib (2.1.1)
 
 PLATFORMS
   x86_64-linux
 
 DEPENDENCIES
+  amazing_print
   fileutils
   json
   oj
   pathname
   rmxp_extractor!
   ruby-progressbar
+  toml-rb
+  yaml
   zlib
 
 BUNDLED WITH

README.md

@@ -1,25 +1,30 @@
 # RMXP-Extractor
 
-A tool for extracting rxdata to a more usable format, namely JSON. You could probably get it working with XML more easily.
-I made this a while back, so I don't really remember how it works, other than it was pretty clunky and relied heavily on iteration.
+A tool designed to dump rxdata to various git compatible formats. Some work better than others, though.
+Toml is extremely slow in comparison to json or yaml. I'd suggest sticking to JSON generally.
+Originally, I used a complex monkey patched system to dump things to a hash that was very slow and imprecise. Now, it uses 
+some neat tricks to do it relatively efficiently. Only problem is, classes dumped or loaded dump their instance variables, meaning that
+classes are **not** initialized with their default values at all. 
+This is important if you are handwriting the config files for whatever reason.
 
-Usage:
-
-`rmxp_extractor import|export|sccripts`
+Pretty print is also a format, but it uses eval for loading. There's likely a better way of doing this out there, but eh, I'm not bothered.
+It's fairly readable though.
 
-# Import
-Import will import all files in Data_JSON into the rxdata format. It'll take a bit.
+There is a minor problem right now with string encoding, especially with oneshot since there's some foul text that's encoded weirdly. 
+I'll try to fix that later.
 
-# Export
-Export will export all files in Data into json. It's a bit faster than JSON.
+Usage:
 
+`rmxp_extractor import | export <type> | scripts"`
 # Script
 
 Allows you to export Scripts/xScripts to a specified folder. You can also import said specified folder back into Scripts/xScripts.
 The last argument `[x]` is optional. Placing just `x` there will extract Scripts.
 
-# Is it flawless
+# Is it flawless?
 
-Running export followed by inport should produce an almost identical file with some minor differences. 
-Move routes will display as blank since they have some extra serialization I don't understand yet, although they still work nontheless.
-You may or may not find some files are also a few bytes longer, I'm unsure why. RPG Maker XP probably doesn't exactly follow Marshal spec and cuts some corners.
+Running export followed by import should produce an almost identical file with some minor differences. 
+As far as I'm aware from my testing, a file diff may say the the two are different, but loading them via marshal provides the exact same instance give or take.
+Ruby reports the original and imported instances as being different when loaded from Marshal, but pretty printing each instance to a file shows this is not the case.
+RPG Maker XP probably doesn't exactly follow Marshal spec and cuts some corners, which is why the file diff shows up differently.
+A diff check tool is provided if you want to try it yourself.

diff_check.rb

@@ -0,0 +1,13 @@
+require_relative "lib/rmxp_extractor/classnames"
+require "amazing_print"
+
+f = File.open("Data_OLD/Map266.rxdata")
+ins1 = Marshal.load(f)
+
+f2 = File.open("Data/Map266.rxdata")
+ins2 = Marshal.load(f2)
+
+f = File.open("ins1", "w")
+f2 = File.open("ins2", "w")
+f.puts ins1.ai(plain: true, raw: true, object_id: false)
+f2.puts ins2.ai(plain: true, raw: true, object_id: false)

lib/rmxp_extractor.rb

@@ -4,29 +4,40 @@ module RMXPExtractor
   require "rmxp_extractor/script_handler"
   require "rmxp_extractor/version"
 
+  FORMATS = ["toml", "json", "yaml", "rb"]
+
   def self.usage
-    STDERR.puts "usage: rmxp_extractor import|export|scripts"
+    STDERR.puts "usage: rmxp_extractor import/export <type = json> | scripts"
     exit 1
   end
 
   def self.process(type)
     RMXPExtractor.usage if type.length < 1
 
-    if type[0] == "import"
-      RMXPExtractor.import
-    elsif type[0] == "export"
-      RMXPExtractor.export
-    elsif type[0] == "scripts"
-      if type[3] == "x"
-        RMXPExtractor.rpgscript(type[2], type[1], true)
-      elsif type.length < 3 || type.length > 4
-        STDERR.puts "usage: rmxp_extractor scripts scripts_dir game_dir [x]"
+    case type[0]
+    when "import"
+      check_format(type[1])
+      import(type[1])
+    when "export"
+      check_format(type[1])
+      export(type[1])
+    when "scripts"
+      if type.length < 4 || type.length > 5
+        STDERR.puts "usage: rmxp_extractor scripts scripts_dir scripts_name game_dir [x]"
         exit 1
-      elsif type[3] == nil
-        RMXPExtractor.rpgscript(type[2], type[1], false)
+      else
+        RMXPExtractor.rpgscript(type[3], type[4], type[5] == "x")
       end
     else
       RMXPExtractor.usage
     end
   end
+
+  def self.check_format(format)
+    format = "json" if format.nil?
+    unless FORMATS.include?(format)
+      warn "Allowed formats: #{FORMATS.to_s}"
+      exit 1
+    end
+  end
 end

lib/rmxp_extractor/classnames.rb

@@ -9,11 +9,11 @@ def create_from_rmxp_serialize(obj)
     ret = {}
     # Hashes have no map method. Why? I dunno!
     obj.each do |key, value|
-      if key.start_with?("class ")
+      if key.is_a?(String) && key.start_with?("class ")
         const = const_from_string(key.delete_prefix("class "))
         ret = const.new.from_rmxp_serialize(value)
       else
-        if key =~ /\A[0-9]+\z/ # Key is an integer
+        if key.is_a?(String) && key =~ /\A[0-9]+\z/ # Key is an integer
           key = key.to_i
         end
         ret[key] = create_from_rmxp_serialize(value)
@@ -108,6 +108,20 @@ def _dump(limit)
   def self._load(obj)
     data = obj.unpack("VVVVVv*")
     @num_of_dimensions, @xsize, @ysize, @zsize, @num_of_elements, *@elements = *data
+    if @num_of_dimensions > 1
+      if @xsize > 1
+        @elements = @elements.each_slice(@xsize).to_a
+      else
+        @elements = @elements.map { |element| [element] }
+      end
+    end
+    if @num_of_dimensions > 2
+      if @ysize > 1
+        @elements = @elements.each_slice(@ysize).to_a
+      else
+        @elements = @elements.map { |element| [element] }
+      end
+    end
   end
 end
 

lib/rmxp_extractor/data_export.rb

@@ -1,9 +1,12 @@
 module RMXPExtractor
-  def self.export
+  def self.export(format)
     STDERR.puts "No Data Directory!" unless Dir.exists? "./Data"
     exit 1 unless Dir.exists? "./Data"
 
     require "json"
+    require "toml-rb"
+    require "yaml"
+    require "amazing_print"
     require "ruby-progressbar"
     require "fileutils"
     require "pathname"
@@ -12,18 +15,19 @@ def self.export
     require_relative "script_handler"
 
     window_size = 120
+    progress_format = "%a /%e |%B| %p%% %c/%C %r files/sec %t, currently processing: "
     progress = ProgressBar.create(
-      format: "%a /%e |%B| %p%% %c/%C %r files/sec %t",
+      format: progress_format,
       starting_at: 0,
       total: nil,
       output: $stderr,
       length: window_size,
-      title: "Exported",
+      title: "exported",
       remainder_mark: "\e[0;30m█\e[0m",
       progress_mark: "█",
       unknown_progress_animation_steps: ["==>", ">==", "=>="],
     )
-    Dir.mkdir "./Data_JSON" unless Dir.exists? "./Data_JSON"
+    Dir.mkdir "./Data_#{format.upcase}" unless Dir.exists? "./Data_#{format.upcase}"
     paths = Pathname.glob(("./Data/" + ("*" + ".rxdata")))
     count = paths.size
     progress.total = count
@@ -32,19 +36,27 @@ def self.export
 
       name = path.basename(".rxdata")
       rxdata = Marshal.load(path.read(mode: "rb"))
-      #puts name.to_s
+      progress.format = progress_format + name.to_s
       case name.to_s
-      when "xScripts"
-        RMXPExtractor.rpgscript("./", "./Scripts", true)
-        content[:version] = VERSION
+      when "xScripts", "Scripts"
+        RMXPExtractor.rpgscript("./", "./Scripts", "#{name.to_s}.rxdata", true)
+        content["version"] = VERSION
       else
-        content[:data] = rxdata.rmxp_serialize
-        content[:version] = VERSION
+        content["data"] = rxdata.rmxp_serialize
+        content["version"] = VERSION
       end
 
-      json = File.open("./Data_JSON/" + name.sub_ext(".json").to_s, "wb")
-      #puts content
-      json.puts JSON.pretty_generate(content)
+      file = File.open("./Data_#{format.upcase}/" + name.sub_ext(".#{format}").to_s, "wb")
+      file.puts case format
+                when "yaml"
+                  content.to_yaml
+                when "json"
+                  JSON.pretty_generate(content)
+                when "toml"
+                  TomlRB.dump(content)
+                when "rb"
+                  content.ai(index: false, indent: 2, plain: true)
+                end
 
       progress.increment
     end

lib/rmxp_extractor/data_import.rb

@@ -1,9 +1,13 @@
 module RMXPExtractor
-  def self.import
-    STDERR.puts "No Data_JSON Directory!" unless Dir.exists? "./Data_JSON"
-    exit 1 unless Dir.exists? "./Data_JSON"
+  def self.import(format)
+    STDERR.puts "No Data_JSON Directory!" unless Dir.exists? "./Data_#{format.upcase}"
+    exit 1 unless Dir.exists? "./Data_#{format.upcase}"
 
     require "oj"
+    require "toml-rb"
+    require "yaml"
+    # require "active_support"
+    # require "active_support/core_ext"
     require "ruby-progressbar"
     require "fileutils"
     require "pathname"
@@ -12,34 +16,46 @@ def self.import
     require_relative "script_handler"
 
     window_size = 120
+    progress_format = "%a /%e |%B| %p%% %c/%C %r files/sec %t, currently processing: "
     progress = ProgressBar.create(
-      format: "%a /%e |%B| %p%% %c/%C %r files/sec %t",
+      format: progress_format,
       starting_at: 0,
       total: nil,
       output: $stderr,
       length: window_size,
-      title: "Imported",
+      title: "imported",
       remainder_mark: "\e[0;30m█\e[0m",
       progress_mark: "█",
       unknown_progress_animation_steps: ["==>", ">==", "=>="],
     )
     Dir.mkdir "./Data" unless Dir.exists? "./Data"
-    paths = Pathname.glob(("./Data_JSON/" + ("*" + ".json")))
+    paths = Pathname.glob(("./Data_#{format.upcase}/" + ("*" + ".#{format}")))
     count = paths.size
     progress.total = count
     paths.each_with_index do |path, i|
-      name = path.basename(".json")
-      json = Oj.load path.read(mode: "rb")
+      name = path.basename(".#{format}")
+      progress.format = progress_format + name.to_s
+      file_contents = path.read(mode: "rb")
+      hash = case format
+        when "json"
+          Oj.load file_contents
+        when "yaml"
+          YAML.load file_contents
+        when "toml"
+          TomlRB.parse file_contents
+        when "rb"
+          eval file_contents # Yes, this SHOULD work. Is it bad? Yes.
+        end
 
-      puts "\n\e[33;1mWARNING: Incompatible version format in #{name.to_s}!\e[0m\n" if json["version"] != VERSION
+      warn "\n\e[33;1mWARNING: Incompatible version format in #{name.to_s}!\e[0m\n" if hash["version"] != VERSION
 
       case name.to_s
-      when "xScripts"
-        RMXPExtractor.rpgscript("./", "./Scripts")
+      when "xScripts", "Scripts"
+        RMXPExtractor.rpgscript("./", "./Scripts", "#{name.to_s}")
         progress.increment
         return
       else
-        content = create_from_rmxp_serialize(json["data"])
+        content = create_from_rmxp_serialize(hash["data"])
       end
 
       rxdata = File.open("./Data/" + name.sub_ext(".rxdata").to_s, "wb")

lib/rmxp_extractor/script_handler.rb

@@ -2,27 +2,15 @@
 require "zlib"
 
 module RMXPExtractor
-  def self.rpgscript(game_dir, scripts_dir, extract = false)
+  def self.rpgscript(game_dir, scripts_dir, script_name, extract = false)
     # Determine version of game engine
     game_data_dir = File.join(game_dir, "Data")
     unless Dir.exist? game_data_dir
       STDERR.puts "error: #{game_dir} does not have a Data subdirectory"
       exit 1
     end
 
-    target_path = nil
-    Dir.entries(game_data_dir).each do |e|
-      ext = File.extname(e)
-      if ext =~ /\.r[xv]data2?/
-        target_path = File.join(game_data_dir, "xScripts" + ext)
-        break
-      end
-    end
-
-    unless target_path
-      STDERR.puts "warning: could not determine game engine version, assuming XP"
-      target_path = File.join(game_data_dir, "xScripts.rxdata")
-    end
+    target_path = File.join(game_data_dir, script_name)
 
     # Generate path of script list
     list_path = File.join(scripts_dir, "_scripts.txt")