Saving tungsten.run post metadata when downloading by user profile

[ConBreezy]

I want to share a problem I encountered, and the workaround I came up with, to see if anyone has suggestions on how I could improve my workaround, or direct me to a better solution. Note: The examples I've included from tungsten.run are SFW(ish), but tungsten.run contains a lot of NSFW content, so be careful if you visit it.

Problem

When I download AI generated images from tungsten.run I also download the metadata, because it's helpful to know the generation data used to create the image. E.g. prompt, negative prompt, sampler, steps, cfg, seed, etc. That desired metadata is included when downloading using the TungstenPostExtractor, i.e. downloading a single image by passing gallery-dl the post url. The problem I encountered, none of that generation data is included in the metadata when using the TungstenUserExtractor, i.e. downloading all images posted by a user by passing gallery-dl the user's profile url.

Here are metadata examples from both extractors.

TungstenUserExtractor Metadata

{
    "uuid": "GVJa9hnvbJua2qE4",
    "title": "Angryven🤣🤣",
    "original_url": "https://api.tungsten.run/v1/upload/RZVY3Qr3yQZmVbM2",
    "original_width": 832,
    "original_height": 1216,
    "resized_url": "https://api.tungsten.run/v1/upload/4TQXjHyLrXJt2mV3",
    "resized_width": 640,
    "resized_height": 936,
    "nsfw": false,
    "user": {
        "uuid": "4994UzqbvLeDPjQc",
        "type": "regular",
        "username": "Luscius",
        "display_name": "Luscius",
        "bio": null,
        "avatar_url": "https://api.tungsten.run/v1/avatar/512x512/4994UzqbvLeDPjQc/1767034821",
        "sm_avatar_url": "https://api.tungsten.run/v1/avatar/64x64/4994UzqbvLeDPjQc/1767034821",
        "is_authorized_uploader": false,
        "created_at": "2025-12-28T23:11:59Z"
    },
    "created_at": "2025-12-30T23:46:33Z",
    "like_count": 13,
    "view_count": 0,
    "comment_count": 0,
    "rating": "neutral",
    "date": "2025-12-30 23:46:33",
    "filename": "RZVY3Qr3yQZmVbM2",
    "extension": "webp",
    "category": "tungsten",
    "subcategory": "user",
    "search_tags": "",
}

TungstenPostExtractor Metadata

{
    "uuid": "GVJa9hnvbJua2qE4",
    "title": "Angryven🤣🤣",
    "description": null,
    "original_url": "https://api.tungsten.run/v1/upload/RZVY3Qr3yQZmVbM2",
    "original_width": 832,
    "original_height": 1216,
    "resized_url": "https://api.tungsten.run/v1/upload/4TQXjHyLrXJt2mV3",
    "resized_width": 640,
    "resized_height": 936,
    "nsfw": false,
    "user": {
        "uuid": "4994UzqbvLeDPjQc",
        "type": "regular",
        "username": "Luscius",
        "display_name": "Luscius",
        "bio": null,
        "avatar_url": "https://api.tungsten.run/v1/avatar/512x512/4994UzqbvLeDPjQc/1767034821",
        "sm_avatar_url": "https://api.tungsten.run/v1/avatar/64x64/4994UzqbvLeDPjQc/1767034821",
        "is_authorized_uploader": false,
        "created_at": "2025-12-28T23:11:59Z"
    },
    "created_at": "2025-12-30T23:46:33Z",
    "tags": [],
    "like_count": 13,
    "view_count": 0,
    "comment_count": 0,
    "rating": "neutral",
    "is_tweakable": true,
    "generation_type": "sdxl",
    "generation_data": {
        "model_version_uuid": "GWFwfBGPtY",
        "prompt": "vanripper (style), 3d rod!, masterpiece, best quality, perfect face, pretty girl, 1girl, alone, brunette, glossy lips, full lips, long eyelashes, wavy hair, pk_venusbody, shiny skin, rav1, grey skin, purple eyes, purple hair, forehead jewel, short hair, cape, hood, leotard, belt, boots, wide hips, huge breasts, crossed arms, HD, looking at viewer, angry, (blush:0.8), (purple lipstick:0.9), black background",
        "negative_prompt": "bad quality, worst quality, worst detail, sketch, censor, artist name, signature, watermark, patreon username, multiple angles, loli, muscular, lipstick, tanline, vibrant colors, no artline, bad anatomy, bug, bad texture",
        "num_images": 4,
        "sampler": "euler_ancestral",
        "steps": 20,
        "cfg": 6.5,
        "clip_skip": 2,
        "seed": 1922531648,
        "width": 832,
        "height": 1216,
        "embeddings": [],
        "loras": [
            {
                "model_version_uuid": "5nMpuV3A5g",
                "weight": 1
            }
        ],
        "controlnets": [],
        "img2img": null,
        "vae": null,
        "postprocess": []
    },
    "generation_postprocessing": null,
    "model_versions": [
        {
            "uuid": "GWFwfBGPtY",
            "name": "V8",
            "model_uuid": "JmTDs9qfK7",
            "model_name": "AniCoreXLIllustrious",
            "model_base": "ilxl",
            "model_type": "checkpoint",
            "model_resolved_nsfw_score": 0,
            "fullname": "AniCoreXLIllustrious - V8",
            "status": "approved",
            "created_at": "2025-11-22T05:20:01Z",
            "cover_url": "https://api.tungsten.run/v1/model_version_cover/GWFwfBGPtY/1763788801",
            "sm_cover_url": "https://api.tungsten.run/v1/model_version_cover/GWFwfBGPtY/1763788801",
            "extra": {
                "dtype": "fp16"
            }
        },
        {
            "uuid": "5nMpuV3A5g",
            "name": "v1",
            "model_uuid": "zvjfMdb7EN",
            "model_name": "Raven | Teen Titans 2003 | DC (Illustrious)",
            "model_base": "ilxl",
            "model_type": "lora",
            "model_resolved_nsfw_score": 0,
            "fullname": "Raven | Teen Titans 2003 | DC (Illustrious) - v1",
            "status": "approved",
            "created_at": "2025-08-05T19:25:26Z",
            "cover_url": "https://api.tungsten.run/v1/model_version_cover/5nMpuV3A5g/1754421927",
            "sm_cover_url": "https://api.tungsten.run/v1/model_version_cover/5nMpuV3A5g/1754421927",
            "extra": {
                "trigger_words": [
                    "rav1",
                    "leotard",
                    "hood",
                    "cape",
                    "short hair",
                    "purple hair",
                    "purple eyes",
                    "grey skin",
                    "forehead jewel",
                    "belt"
                ]
            }
        }
    ],
    "stashes": [],
    "upload_inputs": [],
    "date": "2025-12-30 23:46:33",
    "filename": "RZVY3Qr3yQZmVbM2",
    "extension": "webp",
    "category": "tungsten",
    "subcategory": "post",
}

Solution

I solved this by adding several postprocessors for the "user" subcategory, including the use of a PowerShell script.

1. Use the {uuid} to recreate the post URL https://tungsten.run/post/{uuid}, and store it in the metadata file.

"user": {
				"postprocessors": [
					{
						"name": "metadata",
						"event": "prepare",
						"mode": "modify",
						"fields": {
							"PostURL": "https:\/\/tungsten.run\/post\/{uuid}"
						}
					},
					{
						"name": "metadata",
						"event": "file",
						"filename": "{date:%Y%m%d}-{title!l:?/-/}{uuid}.json",
						"directory": "metadata",
						"mode": "json"
					}

2. Create a PowerShell script to pull the post URL from each metadata file, and write the URLs to a file.

[CmdletBinding()]
param(
    [Parameter(Mandatory=$true)]
    [string]$Metadata,
    
    [Parameter(Mandatory=$true)]
    [string]$OutputFile
)

# Main script logic starts here
$results = Get-ChildItem -Path $Metadata -Recurse -Filter '*.json'  | ForEach-Object {
	$filepath = $_.FullName
	$json = Get-Content -Path $filepath | ConvertFrom-Json
	$json.PostURL | Out-File -FilePath $OutputFile -Encoding ascii -Width 60 -Append
}

3. After all images are downloaded, use the exec postprocessor to execute two commands.
   1. Run the script, passing it the paths for the metadata directory and post URL output file.
   2. Pass the URL file to gallery-dl to download the post metadata but not re-download the images.

{
	"name": "exec",
	"event": "finalize",
	"commands": [
		["C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "-NoProfile", "-ExecutionPolicy", "ByPass", "-Command", "D:\\Redact\\Redact\\Redact\\gallery-dl\\GetPostURLs.ps1 -Metadata {_directory}\\metadata -OutputFile {_directory:[4:]}\\metadata\\PostURLs.txt"],
		["D:\\Redact\\Redact\\Redact\\gallery-dl\\gallery-dl.exe", "--input-file-delete", "{_directory:[4:]}\\metadata\\PostURLs.txt", "--no-skip", "--no-download"]
	]
}

And that's it! All of the images are downloaded from a Tungsten user's profile, and the metadata files created by the TungstenUserExtractor are overwritten by metadata from the TungstenPostExtractor.

Here is my complete Tungsten specific extractor config. (Includes "post" subcategory postprocessors, but that's not relevant to this.)

Tungsten site-specific extractor

"tungsten":
{
	"sleep": "0.5-1.5",
	"sleep-request": "0.5-1.5",
    "filename": "{date:%Y%m%d}-{title!l:?/-/}{uuid}.{extension}",
	"post": {
		"postprocessors": [
			{
				"name": "metadata",
				"event": "init",
				"mode": "modify",
				"fields": {
					"PostURL": "https:\/\/tungsten.run\/post\/{uuid}"
				}
			},
			{
				"name": "metadata",
				"event": "file",
				"filename": "{date:%Y%m%d}-{title!l:?/-/}{uuid}.json",
				"directory": "metadata",
				"mode": "json"
			},
			{
				"name": "mtime"
			}
		]
	},
	"user": {
		"postprocessors": [
			{
				"name": "metadata",
				"event": "prepare",
				"mode": "modify",
				"fields": {
					"PostURL": "https:\/\/tungsten.run\/post\/{uuid}"
				}
			},
			{
				"name": "metadata",
				"event": "file",
				"filename": "{date:%Y%m%d}-{title!l:?/-/}{uuid}.json",
				"directory": "metadata",
				"mode": "json"
			},
			{
				"name": "mtime"
			},
			{
				"name": "exec",
				"event": "finalize",
				"commands": [
					["C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe", "-NoProfile", "-ExecutionPolicy", "ByPass", "-Command", "D:\\Redact\\Redact\\Redact\\gallery-dl\\GetPostURLs.ps1 -Metadata {_directory}\\metadata -OutputFile {_directory:[4:]}\\metadata\\PostURLs.txt"],
					["D:\\Redact\\Redact\\Redact\\gallery-dl\\gallery-dl.exe", "--input-file-delete", "{_directory:[4:]}\\metadata\\PostURLs.txt", "--no-skip", "--no-download"]
				]
			}
		]
	}
}

Notes

• I only have legacy PowerShell 5.1 installed which uses a default encoding of UTF-16LE, which gallery-dl couldn't read. I tried changing the encoding to UTF8, but PowerShell 5.1 doesn't support BOM-less UTF8, which caused gallery-dl to fail reading the first URL of the output file. I didn't feel like installing PowerShell 7, so I just used Ascii encoding instead. That won't be necessary if you have PowerShell 7+.
• When passing the post URL output file path to both commands using {_directory}, I had to strip the first 4 characters {_directory:[4:]}, otherwise the path would be incorrect. I'm not sure why there were extra characters in the directory path.
• I used the --input-file-delete option because the PowerShell script appends URLs to the output file, and I didn't want it to continuously grow every time I download files from the same user.

[mikf]

A slightly simpler way of writing post URLs to a file:

gallery-dl --print-to-file https://tungsten.run/post/{uuid} urls.txt https://tungsten.run/user/USER

When passing the post URL output file path to both commands using {_directory}, I had to strip the first 4 characters {_directory:[4:]}, otherwise the path would be incorrect. I'm not sure why there were extra characters in the directory path.

https://gdl-org.github.io/docs/configuration.html#extractor-path-extended