Skip to content

[ZH] Simulate Replays with CSV files #925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 122 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
8005a7b
Fix messy code and potential null pointer dereference in RecorderClas…
helmutbuhler Apr 13, 2025
c2f8d14
Fix Replay playback not working after running analysis on a replay.
helmutbuhler Apr 13, 2025
942e5e1
Add replay simulation functionality
helmutbuhler Apr 13, 2025
120fa4b
Add -simReplay command line option to run replays without graphics
helmutbuhler Apr 13, 2025
a77e2b3
Return Exitcode 1 if mismatch is detected in a replay while simulating.
helmutbuhler Apr 13, 2025
f0057b7
Make Replay Simulation compile in Release and remove VERIFY_CRC.
helmutbuhler Apr 13, 2025
3241aef
Fix draw assignment in StealthUpdate::changeVisualDisguise()
helmutbuhler Apr 13, 2025
9fa390b
Show analyze and simulate button in replaymenu when DEBUG_LOGGING is …
helmutbuhler Apr 18, 2025
7900ab1
Add code to write out replay list to textfile
helmutbuhler Apr 26, 2025
3499ccd
Update WriteOutReplayList()
helmutbuhler Apr 26, 2025
818161f
Make Replay extra buttons show up always
helmutbuhler Apr 29, 2025
19b8a42
Add m_playbackFrameDuration to Recorder
helmutbuhler Apr 29, 2025
4e562e2
Update WriteOutReplayList and add ReadReplayListFromCsv and -simRepla…
helmutbuhler Apr 29, 2025
60789e6
Add more logging to SimulateReplayList and run each replay in separat…
helmutbuhler Apr 29, 2025
3852327
SimulateReplayList: Count errors
helmutbuhler Apr 29, 2025
8faca18
Factor CreateProcessA into helper function
helmutbuhler Apr 30, 2025
6b23bf8
Make Performance Timers work (WIP)
helmutbuhler May 2, 2025
c0671bd
Make SimulateReplayList faster by calling TheParticleSystemManager->r…
helmutbuhler May 2, 2025
dffca9e
Add GameClient::updateHeadless()
helmutbuhler May 3, 2025
6b3348f
Use pipes to retrieve worker process console output
helmutbuhler May 3, 2025
a2e7c5a
Add SimulateReplayListMultiProcess. It seems to work now!
helmutbuhler May 3, 2025
efd7c14
Use TerminateProcess to work around crash
helmutbuhler May 4, 2025
4673944
Expand columns in WriteOutReplayList and fix ReadLineFromFile
helmutbuhler May 4, 2025
9b7eadc
Add Job stuff to ReplayProcess
helmutbuhler May 4, 2025
a82de61
Fix Jobs for VC6
helmutbuhler May 4, 2025
4835c7f
Revert "Make Performance Timers work (WIP)"
helmutbuhler May 8, 2025
7efcf7c
Merge tag 'maaaaaaaaaaaaain' into sim_replay_sh
helmutbuhler May 8, 2025
7352e5b
Fix date formats in comments
helmutbuhler May 8, 2025
2f01c18
Remove copied comment
helmutbuhler May 8, 2025
08fad6f
Remove bugfix that doesn't belong here
helmutbuhler May 8, 2025
3873f69
Remove added dump code in RecorderClass::handleCRCMessage
helmutbuhler May 8, 2025
eac7e88
Remove merged headless references
helmutbuhler May 8, 2025
c6ede9d
Fix simulation loop in ReplayMenu
helmutbuhler May 9, 2025
8c6bfef
Make ReadReplayListFromCsv work in subfolders
helmutbuhler May 12, 2025
95e6769
Make subprocesses call with -headless
helmutbuhler May 12, 2025
9820db2
Allow more iniCRCs in WriteOutReplayList
helmutbuhler May 12, 2025
1582909
Move Worker Process stuff into separate file
helmutbuhler May 19, 2025
dbeb149
Move command creation out of WorkerProcess::StartProcess
helmutbuhler May 19, 2025
b924e23
Put Replay Simulation Code into own file
helmutbuhler May 22, 2025
8875ac2
Put ReadReplayListFromCsv and WriteOutReplayList into separate file
helmutbuhler May 23, 2025
e493557
Add -jobs commandline
helmutbuhler May 23, 2025
4f59916
Fix whitespace and revert TheFileSystem->getFileListInDirectory fix
helmutbuhler May 23, 2025
1fc78f3
Merge branch 'main' into sim_replay_sh
helmutbuhler May 23, 2025
03bf7b3
Fix wrong merge
helmutbuhler May 23, 2025
7f44033
Don't check for multiple instances in headless mode
helmutbuhler May 23, 2025
0b15614
Add -writeReplayList commandline and make it work with relative paths
helmutbuhler May 24, 2025
417cc01
Use headless mode for -writeReplayList, -simReplay, -simReplayList. R…
helmutbuhler May 24, 2025
208365a
Remove obsolete headless checks
helmutbuhler May 24, 2025
cc1c163
After using simulation button, return to scorescreen instead of Repla…
helmutbuhler May 24, 2025
889fd8e
Remove obsolete headless check in GameLogic::startNewGame
helmutbuhler May 24, 2025
a969df5
Remove Int mismatchFrame simplification in Recorder
helmutbuhler May 24, 2025
cd802df
Remove -writeReplayList and -simReplayList commandline options (goes …
helmutbuhler May 24, 2025
f640caa
Remove "Debug: Simulate Replay"-button in replaymenu (goes into other…
helmutbuhler May 24, 2025
12e00f7
Add -writeReplayList and -simReplayList commandline options
helmutbuhler May 24, 2025
e8b6be5
Remove unnecessary TheControlBar != NULL check
helmutbuhler May 24, 2025
fe00285
Merge branch 'main' into sim_replay_sh
helmutbuhler Jun 5, 2025
1219542
Remove refactoring out of RecorderClass::appendNextCommand()
helmutbuhler Jun 5, 2025
4eb8988
Add stupid const
helmutbuhler Jun 6, 2025
03c530b
Rename Replay Simulation stuff
helmutbuhler Jun 6, 2025
928c624
Fix minor syntax stuff
helmutbuhler Jun 6, 2025
a872b93
Use TheCommandList in RecorderClass::stopPlayback() for consistency
helmutbuhler Jun 6, 2025
c937813
Rename Replay Header member FrameDuration to FrameCount
helmutbuhler Jun 6, 2025
2319046
Make methods of WorkerProcess lowercase
helmutbuhler Jun 6, 2025
8208f9d
Create helper method fetchStdOutput() in WorkerProcess
helmutbuhler Jun 7, 2025
a3c07e9
Some minor changes
helmutbuhler Jun 7, 2025
1b715d2
"-simReplayList" checks belong in other PR
helmutbuhler Jun 7, 2025
115b153
Remove out parameters in WorkerProcess::isDone
helmutbuhler Jun 7, 2025
c2d58ac
variable renaming
helmutbuhler Jun 7, 2025
283e5c6
More renaming
helmutbuhler Jun 7, 2025
8c9612b
remove hacks
helmutbuhler Jun 7, 2025
6a362bd
Add SIMULATE_REPLAYS_SEQUENTIAL and some refactoring
helmutbuhler Jun 7, 2025
1558a3a
more minor stuff
helmutbuhler Jun 7, 2025
568832b
Update comment
helmutbuhler Jun 7, 2025
d90e11a
Move new files to Core
helmutbuhler Jun 7, 2025
ed64972
Merge branch 'main' into sim_replay_sh
helmutbuhler Jun 7, 2025
d660a93
Make -simReplay work without headless
helmutbuhler Jun 14, 2025
020ee42
Add -headless option back
helmutbuhler Jun 14, 2025
b7d559d
Rename -simReplay to -replay
helmutbuhler Jun 14, 2025
702dfa0
Fix merge
helmutbuhler Jun 14, 2025
1e70e82
Merge branch 'main' into sim_replay_sh
helmutbuhler Jun 16, 2025
6ddfdf7
Add m_avoidFirstInstance and m_multiInstance and make -replay work wh…
helmutbuhler Jun 17, 2025
30fc12f
Merge branch 'main' of https://github.com/TheSuperHackers/GeneralsGam…
helmutbuhler Jun 17, 2025
b24eca8
Remove now unused GameMain( int argc, char *argv[] ) parameters
helmutbuhler Jun 17, 2025
d3febea
Changed error handling of -replay commandline parsing to output somet…
helmutbuhler Jun 18, 2025
b3243c8
Allow wildcards in -replay commandline
helmutbuhler Jun 18, 2025
c353b94
Update -replay comment
helmutbuhler Jun 18, 2025
0a63c92
Use mismatchFrame to simplify printf
helmutbuhler Jun 18, 2025
5fced5b
Move m_multiInstance and m_avoidFirstInstance to ClientInstance
helmutbuhler Jun 19, 2025
2d2576f
Take out -multiInstance
helmutbuhler Jun 19, 2025
89431bf
Change comment
helmutbuhler Jun 19, 2025
87462b7
Move CommandLine::parseCommandLineForStartup(); to DebugInit
helmutbuhler Jun 19, 2025
dbf9d82
Convert one missed #if defined(RTS_MULTI_INSTANCE) to use new global
helmutbuhler Jun 19, 2025
a1266af
spelling occured -> occurred
helmutbuhler Jun 19, 2025
6d5b60c
Remove unused var
helmutbuhler Jun 19, 2025
754b054
Add comment
helmutbuhler Jun 19, 2025
a404c92
Merge branch 'sim_replay_sh' into sim_replay_lists_sh
helmutbuhler Jun 19, 2025
7951bc4
Fix wrong automatic merges.
helmutbuhler Jun 19, 2025
104a25b
Add console output in case map is not found
helmutbuhler Jun 19, 2025
a3a21b9
Revert "spelling occured -> occurred" in part
xezon Jun 20, 2025
cb021ff
Refactor ClientInstance code
xezon Jun 20, 2025
a4bf57f
Remove GlobalData::m_showReplayContinueButton
xezon Jun 20, 2025
a0ecf5a
Fix SIMULATE_REPLAYS_SEQUENTIAL constexpr'ness
xezon Jun 20, 2025
ee4a554
Return WorkerProcess::startProcess if CreatePipe has failed
xezon Jun 20, 2025
f80d432
Add assert for m_readHandle
xezon Jun 20, 2025
6b72afd
Clarify -jobs range in comment
xezon Jun 20, 2025
7dbe6f7
Move logic to count running processes into separate function
xezon Jun 20, 2025
ede82f8
Fix constexpr in VC6 build
xezon Jun 20, 2025
118ef4d
Fix debug compile error
xezon Jun 20, 2025
e2947b4
Merge branch 'main' into sim_replay_sh
helmutbuhler Jun 21, 2025
05ca54e
Auto Sync with Generals
helmutbuhler Jun 21, 2025
df74c7f
Apply manual sync for failed autosync
helmutbuhler Jun 21, 2025
1a7d27a
Revert TheCommandList back to TheMessageStream in RecorderClass::stop…
helmutbuhler Jun 21, 2025
06cea29
Merge branch 'sim_replay_sh' into sim_replay_lists_sh
helmutbuhler Jun 22, 2025
6315f27
Merge branch 'main' into sim_replay_lists_sh
helmutbuhler Jun 22, 2025
a417c7a
Fix wrong merge
helmutbuhler Jun 22, 2025
0f9a0cd
Move typename for VS2019 compatibility
helmutbuhler Jun 19, 2025
aaaff0c
More merge fixes
helmutbuhler Jun 22, 2025
c6e803b
Add error checking for ReadReplayListFromCsv and fix up parseSimRepla…
helmutbuhler Jun 19, 2025
b2ab15e
Include mapname in csv file
helmutbuhler Jun 19, 2025
939cda6
Move -writeReplayList to paramsForStartup
helmutbuhler Jun 22, 2025
97d90b7
Add comment
helmutbuhler Jun 22, 2025
609d08b
Revert "Move typename for VS2019 compatibility"
helmutbuhler Jun 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions GeneralsMD/Code/GameEngine/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ set(GAMEENGINE_SRC
Include/Common/RAMFile.h
Include/Common/RandomValue.h
Include/Common/Recorder.h
Include/Common/ReplayListCsv.h
# Include/Common/ReplaySimulation.h
Include/Common/Registry.h
Include/Common/ResourceGatheringManager.h
Expand Down Expand Up @@ -609,6 +610,7 @@ set(GAMEENGINE_SRC
Source/Common/PerfTimer.cpp
Source/Common/RandomValue.cpp
Source/Common/Recorder.cpp
Source/Common/ReplayListCsv.cpp
# Source/Common/ReplaySimulation.cpp
Source/Common/RTS/AcademyStats.cpp
Source/Common/RTS/ActionManager.cpp
Expand Down
2 changes: 2 additions & 0 deletions GeneralsMD/Code/GameEngine/Include/Common/GlobalData.h
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,8 @@ class GlobalData : public SubsystemInterface
std::vector<AsciiString> m_simulateReplays; ///< If not empty, simulate this list of replays and exit.
Int m_simulateReplayJobs; ///< Maximum number of processes to use for simulation, or SIMULATE_REPLAYS_SEQUENTIAL for sequential simulation

AsciiString m_writeReplayList; ///< If not empty, write out list of replays in this subfolder into a csv file (TheSuperHackers @feature helmutbuhler 24/05/2025)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TheSuperHackers epilogue can be omitted for simplicity. It is already marked as such at the Command Line argument entry.


Int m_maxParticleCount; ///< maximum number of particles that can exist
Int m_maxFieldParticleCount; ///< maximum number of field-type particles that can exist (roughly)
WeaponBonusSet* m_weaponBonusSet;
Expand Down
23 changes: 23 additions & 0 deletions GeneralsMD/Code/GameEngine/Include/Common/ReplayListCsv.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/*
** Command & Conquer Generals Zero Hour(tm)
** Copyright 2025 TheSuperHackers
**
** This program is free software: you can redistribute it and/or modify
** it under the terms of the GNU General Public License as published by
** the Free Software Foundation, either version 3 of the License, or
** (at your option) any later version.
**
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
** GNU General Public License for more details.
**
** You should have received a copy of the GNU General Public License
** along with this program. If not, see <http://www.gnu.org/licenses/>.
*/

#pragma once


bool WriteOutReplayList(AsciiString relativeFolder);
bool ReadReplayListFromCsv(AsciiString filename, std::vector<AsciiString>* replayList);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

camelCase function name

47 changes: 47 additions & 0 deletions GeneralsMD/Code/GameEngine/Source/Common/CommandLine.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
#include "Common/Recorder.h"
#include "Common/version.h"
#include "GameClient/ClientInstance.h"
#include "Common/ReplayListCsv.h"
#include "GameClient/TerrainVisual.h" // for TERRAIN_LOD_MIN definition
#include "GameClient/GameText.h"
#include "GameNetwork/NetworkDefs.h"
Expand Down Expand Up @@ -442,6 +443,40 @@ Int parseReplay(char *args[], int num)
return 1;
}

Int parseSimReplayList(char *args[], int num)
{
if (num > 1)
{
AsciiString filename = args[1];
bool success = ReadReplayListFromCsv(filename, &TheWritableGlobalData->m_simulateReplays);
if (!success)
{
printf("Cannot open csv file: \"%s\"\n", filename.str());
exit(1);
}
TheWritableGlobalData->m_playIntro = FALSE;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code starting from here is identical to the one in the function above. This can be refactored to reuse under a common name, for example "prepareClientForReplaySimulation()".

TheWritableGlobalData->m_afterIntro = TRUE;
TheWritableGlobalData->m_playSizzle = FALSE;
TheWritableGlobalData->m_shellMapOn = FALSE;

// Make replay playback possible while other clients (possible retail) are running
rts::ClientInstance::setMultiInstance(TRUE);
rts::ClientInstance::skipPrimaryInstance();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is odd that this preparation for the client is done here in the command line parsing. So for example parsing the repeated -replay command line argument means this setup is being called many times, depending on how often it is used in the command line.

I think this can be safely moved at the end of parseCommandLineForStartup, called as

if (!TheGlobalData->m_replay.empty() || ...)
  prepareClientForReplaySimulation()

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just setting some global variables, no need to make a big thing out of that.

return 2;
}
return 1;
}

Int parseWriteReplayList(char *args[], int num)
{
if (num > 1)
{
TheWritableGlobalData->m_writeReplayList = args[1];
TheWritableGlobalData->m_headless = TRUE;
}
return 1;
}

Int parseJobs(char *args[], int num)
{
if (num > 1)
Expand Down Expand Up @@ -1162,6 +1197,18 @@ static CommandLineParam paramsForStartup[] =
// (If you have 4 cores, call it with -jobs 4)
// If you do not call this, all replays will be simulated in sequence in the same process.
{ "-jobs", parseJobs },

// TheSuperHackers @feature helmutbuhler 28/04/2025
// Pass in a csv file to play back multiple replays. The file must be in the replay folder.
{ "-replayList", parseSimReplayList },
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parseReplayList


// TheSuperHackers @feature helmutbuhler 28/04/2025
// Write out information about all replays in a folder to a csv file.
// Call it with -writeReplayList . for all replays in the replay folder.
// Call it with -writeReplayList folder for all replays in the folder subfolder.
// The result will be saved in replay_list.csv in that folder.
// todo: this is a bit unintuitive. Maybe use ReplaySimulation::resolveFilenameWildcards for this?
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the plan here?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought maybe we combine -writeReplayList with -replay, so you can call it like:
gen.exe -replay rep1.rep -replay fun*.rep -writeReplayList all_fun_and_rep1.csv
But I havn't thought too much about it. I'm open for suggestions.

{ "-writeReplayList", parseWriteReplayList },
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect this lives in the client so that maybe there can also be a debug button in the UI to generate this replay list?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nah, I was just too lazy to make it a separate tool. But we can also add a button for it.

};

// These Params are parsed during Engine Init before INI data is loaded
Expand Down
6 changes: 6 additions & 0 deletions GeneralsMD/Code/GameEngine/Source/Common/GameMain.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@

#include "Common/GameEngine.h"
#include "Common/ReplaySimulation.h"
#include "Common/ReplayListCsv.h"


/**
Expand All @@ -46,6 +47,11 @@ Int GameMain()
{
exitcode = ReplaySimulation::simulateReplays(TheGlobalData->m_simulateReplays, TheGlobalData->m_simulateReplayJobs);
}
else if (!TheGlobalData->m_writeReplayList.isEmpty())
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this live here, inside GameMain, requiring whole GameEngine initialization? Shouldn't this be somewhere else, close to parsing the Command Line?

I suspect TheRecorder and TheMapCache are needed. Perhaps we can just create the systems that are needed instead of creating all of them. Tools also do it that way, for example in MapCacheBuilder.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I initially tried to init those things separately, but they kept piling up. You also need TheLocalFilesystem and I think some more. I think it's just simpler to just delay the writing like this.

{
bool success = WriteOutReplayList(TheGlobalData->m_writeReplayList);
exitcode = success ? 0 : 1;
}
else
{
// run it
Expand Down
2 changes: 2 additions & 0 deletions GeneralsMD/Code/GameEngine/Source/Common/GlobalData.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -960,6 +960,8 @@ GlobalData::GlobalData()
m_simulateReplays.clear();
m_simulateReplayJobs = SIMULATE_REPLAYS_SEQUENTIAL;

m_writeReplayList = "";

for (i = LEVEL_FIRST; i <= LEVEL_LAST; ++i)
m_healthBonus[i] = 1.0f;

Expand Down
Loading
Loading