`--processors` / `n_processors` greater than 1 produces incorrect token timestamps

[rotemdan]

The --processors / n_processors option, which initially I assumed was related to core count, actually seems to split the audio to chunks and process them in parallel.

When n_processors is set to 1, the correct token timestamps are produced for each segment:

{
	"timestamps": {
		"from": "00:00:20,000",
		"to": "00:00:24,560"
	},
	"offsets": {
		"from": 20000,
		"to": 24560
	},
	"text": " actually took a watch out of its waistcoat pocket and look at it and then hurried on,",
	"tokens": [
		{
			"text": " actually",
			"timestamps": {
				"from": "00:00:20,020",
				"to": "00:00:20,530"
			},
			"offsets": {
				"from": 20020,
				"to": 20530
			},
			"id": 1682,
			"p": 0.998559,
			"t_dtw": -1
		},
		{
			"text": " took",
			"timestamps": {
				"from": "00:00:20,530",
				"to": "00:00:20,760"
			},
			"offsets": {
				"from": 20530,
				"to": 20760
			},
			"id": 1718,
			"p": 0.999518,
			"t_dtw": -1
		},
		{
			"text": " a",
			"timestamps": {
				"from": "00:00:20,810",
				"to": "00:00:20,850"
			},
			"offsets": {
				"from": 20810,
				"to": 20850
			},
			"id": 257,
			"p": 0.999047,
			"t_dtw": -1
		},

When n_processors is set to 2, the timestamps reset at 20.930s (which is possibly the split point used):

{
	"timestamps": {
		"from": "00:00:20,930",
		"to": "00:00:25,970"
	},
	"offsets": {
		"from": 20930,
		"to": 25970
	},
	"text": " watch out of its waistcoat pocket and look at it and then hurry on, Alice started to her feet,",
	"tokens": [
		{
			"text": "[_BEG_]",
			"timestamps": {
				"from": "00:00:00,000",
				"to": "00:00:00,000"
			},
			"offsets": {
				"from": 0,
				"to": 0
			},
			"id": 50363,
			"p": 0.99157,
			"t_dtw": -1
		},
		{
			"text": " watch",
			"timestamps": {
				"from": "00:00:00,000",
				"to": "00:00:00,330"
			},
			"offsets": {
				"from": 0,
				"to": 330
			},
			"id": 2342,
			"p": 0.641946,
			"t_dtw": -1
		},
		{
			"text": " out",
			"timestamps": {
				"from": "00:00:00,330",
				"to": "00:00:00,530"
			},
			"offsets": {
				"from": 330,
				"to": 530
			},
			"id": 503,
			"p": 0.99322,
			"t_dtw": -1
		},
		{
			"text": " of",
			"timestamps": {
				"from": "00:00:00,530",
				"to": "00:00:00,660"
			},
			"offsets": {
				"from": 530,
				"to": 660
			},
			"id": 286,
			"p": 0.997212,
			"t_dtw": -1
		},

To work around this problem, it's possible, for the special case of processors > 1, to maybe try to track the [_BEG_] tokens which appear to 'zero-out' the time and add some offsets relative to them, but that makes things a bit more complex than needed.

In general, it would be more natural and easy to just have the tokens timed correctly relative to the source audio, rather than needing to deal with complex hacks trying to guess the intended timestamps.

[hlevring]

hmm ..this is an eye opener, I always thought that whisper.cpp only supported the sequential input form.

[sarthak-sehgal]

@rotemdan @ggerganov does this issue still exist? If so, it would be great to have some instructions to reproduce. I'd like to get started with whisper-cpp and thought this would be a good first issue. I tried to reproduce it and seem to get correct results. Command ran on a sample 30s audio: ./build/bin/whisper-cli -f samples/mysample.mp3 --max-len 10 --output-json --processors 2 --duration 30000

Output:

whisper_full_parallel: the audio has been split into 2 chunks at the following times:
whisper_full_parallel: split 1 - 00:11:42.570

I see that the timestamps are correct at the split boundary:

{
        "timestamps": {
                "from": "00:11:41,760",
                "to": "00:11:42,340"
        },
        "offsets": {
                "from": 701760,
                "to": 702340
        },
        "text": " long?"
},
{
        "timestamps": {
                "from": "00:11:42,570",
                "to": "00:11:42,980"
        },
        "offsets": {
                "from": 702570,
                "to": 702980
        },
        "text": " And he"
},
{
        "timestamps": {
                "from": "00:11:42,980",
                "to": "00:11:43,380"
        },
        "offsets": {
                "from": 702980,
                "to": 703380
        },
        "text": " couldn't"
},

[ggerganov]

I think it might have been fixed with #3173.

If it is not longer reproducible, I will close this issue.