Unable to approach loss of less than 0.7 even when testing multiple learning rates.

[maxisme]

I have tried many different learning rates and optimizers but I have not once seen a min loss drop below 0.69.

If I use learning_rate = 1e-2:

iter:    20, loss min|avg|max: 0.713|2.607|60.013, batch-p@3: 4.43%, ETA: 6:01:18 (0.87s/it)
iter:    40, loss min|avg|max: 0.696|1.204|21.239, batch-p@3: 5.99%, ETA: 5:58:53 (0.86s/it)
iter:    60, loss min|avg|max: 0.696|1.643|25.543, batch-p@3: 4.69%, ETA: 5:36:32 (0.81s/it)
iter:    80, loss min|avg|max: 0.695|1.679|42.339, batch-p@3: 7.03%, ETA: 5:58:01 (0.86s/it)
iter:   100, loss min|avg|max: 0.694|1.806|47.572, batch-p@3: 6.51%, ETA: 6:08:57 (0.89s/it)
iter:   120, loss min|avg|max: 0.695|1.200|21.791, batch-p@3: 4.43%, ETA: 6:14:15 (0.90s/it)
iter:   140, loss min|avg|max: 0.694|2.744|87.940, batch-p@3: 5.47%, ETA: 6:21:29 (0.92s/it)

If I use learning_rate = 1e-6:

iter:    20, loss min|avg|max: 0.741|14.827|440.151, batch-p@3: 1.04%, ETA: 6:23:26 (0.92s/it)
iter:    40, loss min|avg|max: 0.712|9.662|146.125, batch-p@3: 2.86%, ETA: 6:03:24 (0.87s/it)
iter:    60, loss min|avg|max: 0.697|3.944|100.707, batch-p@3: 4.17%, ETA: 6:10:44 (0.89s/it)
iter:    80, loss min|avg|max: 0.695|2.408|75.002, batch-p@3: 2.86%, ETA: 5:44:48 (0.83s/it)
iter:   100, loss min|avg|max: 0.694|2.272|67.504, batch-p@3: 2.86%, ETA: 6:03:45 (0.88s/it)
iter:   120, loss min|avg|max: 0.694|1.091|17.292, batch-p@3: 2.86%, ETA: 5:42:45 (0.83s/it)
iter:   140, loss min|avg|max: 0.693|1.069|15.975, batch-p@3: 5.73%, ETA: 5:46:48 (0.84s/it)
...
iter:   900, loss min|avg|max: 0.693|0.694| 0.709, batch-p@3: 2.08%, ETA: 5:15:00 (0.78s/it)
iter:   920, loss min|avg|max: 0.693|0.693| 0.701, batch-p@3: 2.34%, ETA: 5:39:12 (0.85s/it)
iter:   940, loss min|avg|max: 0.693|0.694| 0.704, batch-p@3: 5.99%, ETA: 5:46:12 (0.86s/it)
iter:   960, loss min|avg|max: 0.693|0.693| 0.705, batch-p@3: 2.86%, ETA: 5:24:59 (0.81s/it)
iter:   980, loss min|avg|max: 0.693|0.693| 0.700, batch-p@3: 3.65%, ETA: 5:39:47 (0.85s/it)
iter:  1000, loss min|avg|max: 0.693|0.693| 0.698, batch-p@3: 3.39%, ETA: 5:27:59 (0.82s/it)
iter:  1020, loss min|avg|max: 0.693|0.693| 0.700, batch-p@3: 6.51%, ETA: 5:36:38 (0.84s/it)
iter:  1040, loss min|avg|max: 0.693|0.694| 0.699, batch-p@3: 2.86%, ETA: 5:22:05 (0.81s/it)
...
iter:  1640, loss min|avg|max: 0.693|0.693| 0.694, batch-p@3: 2.60%, ETA: 5:09:58 (0.80s/it)
iter:  1660, loss min|avg|max: 0.693|0.693| 0.694, batch-p@3: 2.08%, ETA: 5:48:27 (0.90s/it)
iter:  1680, loss min|avg|max: 0.693|0.693| 0.694, batch-p@3: 4.43%, ETA: 5:23:23 (0.83s/it)
iter:  1700, loss min|avg|max: 0.693|0.693| 0.694, batch-p@3: 6.51%, ETA: 5:25:04 (0.84s/it)
iter:  1720, loss min|avg|max: 0.693|0.693| 0.694, batch-p@3: 3.12%, ETA: 5:39:08 (0.87s/it)

What does this affectively mean? "Nonzero triplets never decreases" - not quite sure what that means?

---

I am using the vgg dataset with the file structure like this:

class_a/file.jpg
class_b/file.jpg
class_c/file.jpg
...

I set the pids, fids = [], [] like this:

classes = [path for path in os.listdir(DATA_DIR) if os.path.isdir(os.path.join(DATA_DIR, path))]
for c in classes:
    for file in glob.glob(DATA_DIR+c+"/*.jpg"):
        pids.append(c)
        fids.append(file)

where DATA_DIR is the directory of the vgg dataset.

[lucasb-eyer]

Hi, I have seen the VGG-Face2 dataset and have always wanted to train on it, but never found the time, unfortunately. So far, I was almost always able to train on any dataset I have tried, but for a few especially difficult ones, it took considerable time to find good hyperparameters. (I think for the worst case, only a single value for hyperparameters converged!)

You might also want to try another optimizer or, for Adam, it can be necessary to also tune the "epsilon" value, as mentioned in the TensorFlow documentation.

An option to make training more robust, which has been done in many recent papers, is to add a softmax loss to the total loss, this usually helps overcome the "difficult phase" which is what you see.

Finally, this is also what happens when you make a mistake somewhere, for example mix-up PIDs by mistake, or pre-process images so as to make them corrupt, or whatever other mistakes can happen that kill the structure in the data.

[maxisme]

Yes the latter is what I am very worried about. Do you know if there any way to efficiently output an anchor, positive and negative at some point during training? I am new to TF. I have tried the other optimiser you mentioned in the paper no difference. I am judging that the training is working on the fact that if the min loss drops below 0.69 early on for efficiency sake is this a really poor idea?

Thank you very much for getting back to me!

---

currently trying facenets:

optimizer = tf.train.RMSPropOptimizer(learning_rate, decay=0.9, momentum=0.9, epsilon=1.0)

[lucasb-eyer]

For completeness, I'm repeating here what I wrote in an e-mail to you. For especially difficult datasets, you really need to try many hyperparameter values to find some that converge. For example, try learning rates 1e-1, 3e-2, 1e-2, 3e-3, ..., 1e-8, each of them for 2-5k updates, and stop early for those that don't go below maybe 0.65 or so. That should be feasible in one night or so.

About outputting things, have a look at "summaries" in tensorflow, you can output even images and look at them in tensorboard.

It doesn't make sense to try optimizer settings from other papers, every model/dataset combination has their own sweet spot, so if the paper you take one from doesn't use the same model/dataset as you, it's only luck if the optimizer setting will work or not. However, RMSProp is a good one to try, too.

[maxisme]

Okay after starting the sesh I have ran:

print(sess.run(fids))
print(sess.run(pids))

this returns:

['/path/to/data-set/n006101/0320_01.jpg'
 '/path/to/data-set/n006101/0131_05.jpg'
 '/path/to/data-set/n006101/0043_01.jpg'
 '/path/to/data-set/n006101/0114_01.jpg'
 '/path/to/data-set/n004359/0174_04.jpg'
 '/path/to/data-set/n004359/0058_01.jpg'
 '/path/to/data-set/n004359/0241_01.jpg'
 '/path/to/data-set/n004359/0175_01.jpg'
 '/path/to/data-set/n007003/0417_05.jpg'
 '/path/to/data-set/n007003/0209_01.jpg'
 '/path/to/data-set/n007003/0077_01.jpg'
 '/path/to/data-set/n007003/0057_01.jpg'
 '/path/to/data-set/n000457/0203_01.jpg'
 '/path/to/data-set/n000457/0159_01.jpg'
 '/path/to/data-set/n000457/0197_01.jpg'
 '/path/to/data-set/n000457/0161_01.jpg'
 '/path/to/data-set/n000549/0363_02.jpg'
 '/path/to/data-set/n000549/0141_01.jpg'
 '/path/to/data-set/n000549/0414_01.jpg'
 '/path/to/data-set/n000549/0328_01.jpg'
 '/path/to/data-set/n006797/0147_02.jpg'
 '/path/to/data-set/n006797/0035_02.jpg'
 '/path/to/data-set/n006797/0101_02.jpg'
 '/path/to/data-set/n006797/0145_01.jpg'
 '/path/to/data-set/n001789/0210_01.jpg'
 '/path/to/data-set/n001789/0012_01.jpg'
 '/path/to/data-set/n001789/0087_01.jpg'
 '/path/to/data-set/n001789/0159_01.jpg'
 '/path/to/data-set/n000473/0074_01.jpg'
 '/path/to/data-set/n000473/0039_02.jpg'
 '/path/to/data-set/n000473/0174_01.jpg'
 '/path/to/data-set/n000473/0211_01.jpg'
 '/path/to/data-set/n000489/0008_01.jpg'
 '/path/to/data-set/n000489/0131_02.jpg'
 '/path/to/data-set/n000489/0176_01.jpg'
 '/path/to/data-set/n000489/0221_01.jpg'
 '/path/to/data-set/n002198/0159_07.jpg'
 '/path/to/data-set/n002198/0033_02.jpg'
 '/path/to/data-set/n002198/0181_03.jpg'
 '/path/to/data-set/n002198/0126_01.jpg'
 '/path/to/data-set/n000777/0445_01.jpg'
 '/path/to/data-set/n000777/0126_01.jpg'
 '/path/to/data-set/n000777/0456_04.jpg'
 '/path/to/data-set/n000777/0392_01.jpg'
 '/path/to/data-set/n007482/0196_03.jpg'
 '/path/to/data-set/n007482/0013_01.jpg'
 '/path/to/data-set/n007482/0344_01.jpg'
 '/path/to/data-set/n007482/0064_01.jpg'
 '/path/to/data-set/n005586/0061_02.jpg'
 '/path/to/data-set/n005586/0100_01.jpg'
 '/path/to/data-set/n005586/0144_01.jpg'
 '/path/to/data-set/n005586/0382_01.jpg'
 '/path/to/data-set/n000944/0317_01.jpg'
 '/path/to/data-set/n000944/0144_01.jpg'
 '/path/to/data-set/n000944/0469_01.jpg'
 '/path/to/data-set/n000944/0030_01.jpg'
 '/path/to/data-set/n003644/0695_01.jpg'
 '/path/to/data-set/n003644/0104_02.jpg'
 '/path/to/data-set/n003644/0032_01.jpg'
 '/path/to/data-set/n003644/0131_01.jpg'
 '/path/to/data-set/n006191/0241_01.jpg'
 '/path/to/data-set/n006191/0186_03.jpg'
 '/path/to/data-set/n006191/0073_01.jpg'
 '/path/to/data-set/n006191/0157_05.jpg'
 '/path/to/data-set/n004641/0269_01.jpg'
 '/path/to/data-set/n004641/0030_01.jpg'
 '/path/to/data-set/n004641/0179_01.jpg'
 '/path/to/data-set/n004641/0132_01.jpg'
 '/path/to/data-set/n000881/0086_01.jpg'
 '/path/to/data-set/n000881/0351_03.jpg'
 '/path/to/data-set/n000881/0233_01.jpg'
 '/path/to/data-set/n000881/0130_01.jpg']
['n000203' 'n000203' 'n000203' 'n000203' 'n009265' 'n009265' 'n009265'
 'n009265' 'n006279' 'n006279' 'n006279' 'n006279' 'n005480' 'n005480'
 'n005480' 'n005480' 'n005396' 'n005396' 'n005396' 'n005396' 'n007609'
 'n007609' 'n007609' 'n007609' 'n002699' 'n002699' 'n002699' 'n002699'
 'n008955' 'n008955' 'n008955' 'n008955' 'n000885' 'n000885' 'n000885'
 'n000885' 'n007587' 'n007587' 'n007587' 'n007587' 'n008725' 'n008725'
 'n008725' 'n008725' 'n006369' 'n006369' 'n006369' 'n006369' 'n008052'
 'n008052' 'n008052' 'n008052' 'n000116' 'n000116' 'n000116' 'n000116'
 'n008270' 'n008270' 'n008270' 'n008270' 'n000668' 'n000668' 'n000668'
 'n000668' 'n006747' 'n006747' 'n006747' 'n006747' 'n002827' 'n002827'
 'n002827' 'n002827']

Am I correct in thinking things have gone terribly wrong hence the first two files are not class n000203 etc...?

[lucasb-eyer]

Yes, good find, it seems things have gone wrong in the dataset preparation somewhere!

PS: Thanks for going through moving this to a new issue!

[Pandoro]

You should probably run

print(sess.run([pids, fids]))

instead. The way you call it separately will show you the pids and fids from different batches.

[maxisme]

Awkward. Yeah works fine :( what is the problem!!

[lucasb-eyer]

@Pandoro is right, I rejoiced too early.

[maxisme]

Definitely a good port of call for anyone running into this issue in the future though. I am going to carry on going through the learning_rates but have a feeling this is not the solution. As the variance is so massive you are very likely to hit a number below 0.7 very early on yet this has never happened with ~10 different learning_rates .

[maxisme]

I just set the margin to 10.0 now the min loss is 10. Is that expected? So soft margin is 0.7? and setting margin = 'none' my min loss hits 0 and stays at 0.

[Pandoro]

@maxisme what is happening is that your embeddings are collapsing all to a single point. This is also what we discussed quite extensively in the supplementary material of our paper. When you use a margin based loss, if all embedding vectors collapse to the same point, your loss will always be equal to the margin. If you use the soft-margin loss, that is why you end up with ln(1 + exp(0)) ≈ 0.7 as the value it goes against.

Also make sure you don't confuse "learning rate" with "loss". Those are fundamentally different things.

Nevertheless, I can just second the things @lucasb-eyer suggested in order to fix the collapsing of the embeddings.

[maxisme]

@Pandoro thank you for that explanation. Does "collapsing all to a single point" mean all the values/embeddings/features outputted by the CNN are becoming the same no matter the input data (image) - i.e the network weights are becoming the same? Or have I misunderstood?

[lucasb-eyer]

All embeddings/output values are becoming the same, yes, but that doesn't mean that network weights are becoming the same (that isn't too meaningful of a statement, actually). Please check the paper's appendix.

In addition to what I recommended above (highly recommended to try thoroughly!) please also check if you are actually loading pre-trained weights, that's another common mistake people forget to do!

[maxisme]

I do https://github.com/VisualComputingInstitute/triplet-reid/blob/master/train.py#L286 but I have deleted L365 through L367 as I am not bother about checkpointing. Is that what you mean?