-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathChangeLog
7050 lines (3566 loc) · 198 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# Changelog
## v6.0.1 (2025-01-23)
### Changes
* [update v6.0.1] login lowercase. [terrtia]
* [uodate v6.0.1] add update version. [terrtia]
* [taxomomies + galaxy] update. [terrtia]
* [show tracker] filter result by object type. [terrtia]
* [updater] use tag subversion, ex:v6.0.1. [terrtia]
* [crawler] improve queued error log. [terrtia]
* [retro hunt + update] add ocr retro hunt + new update. [terrtia]
* [user] force lowercase + add v6.0.1 update. [terrtia]
* [chat explorer] show message, open message in chat/subchannel/thread. [terrtia]
* [flask] update Flask_config.py. [Ikko Eltociear Ashimine]
accross -> across
* [chat explorer] show chats tags. [terrtia]
* [flask] set proxy ProxyFix. [terrtia]
* [test] log if flask is reachable. [terrtia]
* [user-account viewer] add messages heatmap by year. [terrtia]
* [chat viewer] heatmap messages, add year selector. [terrtia]
* [chat viewer] add chat messages by current year heatmap. [terrtia]
* [message description] add chat name and username. [terrtia]
* [chat explorer] improve basic chat card, add chat particpants btn + improve btn css. [terrtia]
* [cryptocurrency] add ripple address subtype + correlation. [terrtia]
* [misp taxonomies galaxy] bump. [terrtia]
* [blur unsafe images] blur violence and pornography-illicit-or-illegal tags + add warning message. [terrtia]
* [search domain by name] sanityze domain name + show domain. [terrtia]
* [README] update dashboard image. [terrtia]
### Fix
* [ail updater] remove update between tags. [terrtia]
* [ail updater] fix upper tags list + is_fork. [terrtia]
* [show Domain] unblur default image if domain is down. [terrtia]
* [module extrsctor] log invalid object meta. [terrtia]
* [module extrsctor] fix invalid object meta. [terrtia]
* [retro hunt] fix object to resume. [terrtia]
* [retro hunt] fix item iterator. [terrtia]
* [delete retro hunt] force pause state before delete. [terrtia]
* [retro hunt] fix retro hunt resume. [terrtia]
* [crawler] increase timeout QUEUED captures. [terrtia]
* [crawler] increase timeout QUEUED captures. [terrtia]
* [crawler] timeout QUEUED captures. [terrtia]
* [crawler] title extraction, sigalarm raised by signal.alarm and sleep. [terrtia]
* [crawler] debug signal timeout. [terrtia]
* [crawler] log timeout + debug signal timeout. [terrtia]
* [title beautifullsoup] sigalrm. [terrtia]
* [title beautifullsoup] add signal, BeautifulSoup html.parser is stuck. [terrtia]
* [crawler] crawler queued capture loop. [terrtia]
* [crawler] debug crawler queued capture loop. [terrtia]
* [chat messages by year] fix subchannel nb_max. [terrtia]
* [chat forum] fix get years daterange. [terrtia]
* [chat explorer] fix protocols name list order. [terrtia]
* [crawler] fix reload_crawlers_stats queues stats. [terrtia]
* [exifs module] Mp4 UnidentifiedImageError. [terrtia]
* [investigation] add object with spaces in id to an investigation. [terrtia]
* [module_extractor] fix onion extraction. [terrtia]
* [module_extractor] don't extract onion from crawled items. [terrtia]
* [crawler] avoid crawler loop if a capture end up in an invalid state. [terrtia]
### Other
* Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia]
* Merge pull request #253 from eltociear/patch-1. [Thirion Aurélien]
chg: [flask] update Flask_config.py
* Merge pull request #250 from FafnerKeyZee/patch-1. [Thirion Aurélien]
Update Tracker.py
* Update Tracker.py. [Fafner [_KeyZee_]]
If self._get_field('level') is None, trying to convert it into int will crash the system :)
2024-12-23 14:22:46,232 Flask_server ERROR:Exception on /trackers [GET]
Traceback (most recent call last):
File "/home/user/ail-framework/AILENV/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ail-framework/AILENV/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ail-framework/AILENV/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ail-framework/AILENV/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ail-framework/AILENV/lib/python3.12/site-packages/flask_login/utils.py", line 290, in decorated_view
return current_app.ensure_sync(func)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ail-framework/var/www/Role_Manager.py", line 81, in decorated_view
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ail-framework/var/www/blueprints/hunters.py", line 74, in trackers_dashboard
trackers = Tracker.get_trackers_dashboard(user_org, user_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ail-framework/bin/lib/Tracker.py", line 785, in get_trackers_dashboard
if not tracker.check_level(user_org, user_id):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/ail-framework/bin/lib/Tracker.py", line 220, in check_level
level = self.get_level()
^^^^^^^^^^^^^^^^
File "/home/user/ail-framework/bin/lib/Tracker.py", line 183, in get_level
level = int(self._get_field('level'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Merge pull request #251 from FafnerKeyZee/patch-2. [Thirion Aurélien]
Update abstract_chat_object.py
* Update abstract_chat_object.py. [Fafner [_KeyZee_]]
https://github.com/ail-project/ail-framework/commit/9a388dc9cb05eec9f612343fcdf4579b18bf1e00 broke the Forum view, we can not do stat on forum, only on subchan :)
* Merge pull request #249 from vncloudsco/master. [Alexandre Dulaunoy]
Update Install silent and Dockerfile update
* Update Install silent and dockerfile update. [Mạnh Tường Solo]
## v6.0 (2024-12-09)
### Changes
* [v6.0] add v6.0 update script. [terrtia]
* [pubsublogger] remove old redis_logger. [terrtia]
* [dashboard] filter submission tags. [terrtia]
* [dashboard] show tracker description. [terrtia]
* [queue] save module start time + pid. [terrtia]
* [dashboard] add day date. [terrtia]
* [flask] remove old modules + modules loader. [terrtia]
* [dashboard] add crawler stats. [terrtia]
* [dashboard] add objects tooltip. [terrtia]
* [dashboard] nb objects per days + websocket update. [terrtia]
* [dashboard] add echart feeder graph + cleanup olg graph libs + dashboard, show today nb objects. [terrtia]
* [flask modules] remove old modules. [terrtia]
* [fontawesome] v6.6.0 migration. [terrtia]
* [barcode] add and extract barcode object. [terrtia]
### Fix
* [dashboard] fix day format. [terrtia]
* [dashboard] fix feeeder names. [terrtia]
* [retro hunts] remove objects. [terrtia]
* [retro hunt] fix add tags. [terrtia]
* [object item] fix show item url. [terrtia]
* [barcode] fix message card. [terrtia]
* [show organisation] fix sidebar. [terrtia]
## v5.9 (2024-10-18)
### Changes
* [qrcodes] daterange search: sort qrcode by content. [terrtia]
* [dom-hash] add dom-hash object compute dom-hash for domains and crawled items. [terrtia]
* [CEDetector] tag domains. [terrtia]
* [CEDetector] add detect message. [terrtia]
* [titles] setup Titles queues + CEDetector module. [terrtia]
* [tools] add reprocess title + CEDetector. [terrtia]
* [crawler] disable unsafe title auto tagging. [terrtia]
* [module] debug. [terrtia]
* [module] add CEDetector. [terrtia]
* [crawler] tag domain by vanity. [terrtia]
* [crawler] crawl list urls: filter duplicates. [terrtia]
* [crawler] submit free text of urls to crawl. [terrtia]
* [onion module] filter onion v2. [terrtia]
* [show domain] show titles content + fix item tags. [terrtia]
* [crawler] update tor user agent. [terrtia]
* [message image] show qrcodes extracted. [terrtia]
* [domain lookup] extract domain from url input. [terrtia]
* [api] rename domain lookup. [terrtia]
### Fix
* [module] typo. [terrtia]
* [tag] tag delete: fix update first/last seen. [terrtia]
* [show domain] fix correlation button if nb correl == 0. [terrtia]
* [crawler] filter lookup tags. [terrtia]
* [crawler] filter lookup parent + domain daterange. [terrtia]
## v5.8 (2024-10-03)
### Changes
* [tags search] search qrcode by tags. [terrtia]
* [api] add domain lookup. [terrtia]
* [chats explorer] image lazy loading. [terrtia]
* [favicon] favicons by daterange, lazy loading. [terrtia]
* [qrcode] improve qrcode extractor + add v5.8 update. [terrtia]
* [debug] debug empty qr content. [terrtia]
* [qrcode] extract qrcode content from images and screenshots + qrcode object + correlation. [terrtia]
* [hunter sidebar] allign icons. [terrtia]
* [show item] replace canevas by image blur + add a button to wrap text. [terrtia]
* [create edit user] add org selector search. [terrtia]
* [username] add username search. [terrtia]
* [tags search] show last tag(s) date if no result. [terrtia]
* [show domain screenshots] replace canevas by img blur. [terrtia]
### Fix
* [qrcode extractor] fix catch cv2 exception. [terrtia]
* [qrcode extractor] fix empty qreader. [terrtia]
* [qrcode extractor] filter code39. [terrtia]
* [qrcode extractor] fix image/screenshot tag. [terrtia]
* [bitcoin bech32] fix address validator. [terrtia]
* [qrcode extractor] filter invalid image. [terrtia]
* [qrcode extractor] filter invalid image. [terrtia]
* [qrcode extractor] debug. [terrtia]
* [qrcode extractor] fix invalid jpeg. [terrtia]
* [qrcode install] add missing lib. [terrtia]
* [mail module] fix typo. [terrtia]
* [tags] fix invalid tag obj_id function. [terrtia]
* [crwaler dashboard] fix current day domains up/down links. [terrtia]
* [investigation] fix object table long object id. [terrtia]
* [v5.7 update] add missing python packages. [terrtia]
* [show domain] avoid overlaping table/image. [terrtia]
* [crawler barchart] up/down colors. [terrtia]
## v5.7 (2024-09-17)
### Changes
* [submodule] update. [terrtia]
* [module] disable indexer queue. [terrtia]
* [orgs list] show numbers of users. [terrtia]
* [modules] disable LibInjection. [terrtia]
* [chat request] add other chat type + placeholders. [terrtia]
* [chats] show username. [terrtia]
* [flask] use the same cookie name + cache git commits, tags and branch name. [terrtia]
* [user roles] rename coordinator as org_admin. [terrtia]
* [search] disable whoosh index and search. [terrtia]
* [create user] new account qr code. [terrtia]
* [create user] new account qr code. [terrtia]
* [trackers] check acl of last trackers matches. [terrtia]
* [ocr] ocr card, add first seen and last seen. [terrtia]
* [2fa] add instance name in config. [terrtia]
* [MISP Export] filter None MISP objects. [terrtia]
* [chats explorer] add chat monitoring request. [terrtia]
* [access logs] add user_agent. [terrtia]
* [access logs] add user_agent. [terrtia]
* [UI] abort 403 and 404. [terrtia]
* [cookiejars] show organisation. [terrtia]
* [trackers + retro_hunts] show organisation. [terrtia]
* [investigations] show organisation. [terrtia]
* [UI] replace remote_addr by access_route. [terrtia]
* [user profile] show user organisation. [terrtia]
* [UI + API] update endpoints ACL. [terrtia]
* [acl] refactor acl cookiejars, trackers, retro_hunts, investigation + refactor users roles. [terrtia]
* [update] create default org + update trackers/retro_hunts/investigations creator_org. [terrtia]
* [items] check obj invesigations acl. [terrtia]
* [module extractor] check tracker and retro hunt acl. [terrtia]
* [user] refactor user creation and edit + edit user organisation. [terrtia]
* [trackers + retro_hunts] add org level to retro_hunt + update acl to support org + refactor trackers org. [terrtia]
* [cookiejar] add org level to cookiejar + update acl to support org. [terrtia]
* [ail orgs] refactor organisations + add org level to investigations. [terrtia]
* [core] add users organisation + tracker acl by organisation. [terrtia]
* [logs] add access logs. [terrtia]
* [ail users] cleanup Role_Manager. [terrtia]
* [ail users] save last api usage. [terrtia]
* [tests] improve test output, show all running screens. [terrtia]
* [ail users] remove old User lib + improve API test. [terrtia]
* [crawler] add button to manually delete lacus queues. [terrtia]
* [misp-taxonomies] updated. [Alexandre Dulaunoy]
* [misp-galaxy] updated to the latest version. [Alexandre Dulaunoy]
* [chat viewer] show message forwarded from. [terrtia]
* [usernames timeline] remove empty usernames. [terrtia]
* [chats explorer] filter Chats that don't contain messages. [terrtia]
* [user-account] chat user-accounts: show usernames list + usernames timeline. [terrtia]
* [dashboard] replace interval requests by event stream. [terrtia]
* [update] add v5.7 update. [terrtia]
* [users] manually logout users/kill user session + add users meta:creator, creation date, last_edit, last_login, last_seen, is_logged. [terrtia]
* [2FA] user + admin manage 2FA OTP. [terrtia]
* [login] add 2FA TOTP and HOTP. [terrtia]
* [chord] add parameters to filtersby min and max value. [terrtia]
* [chats viewer] add relationships buttons. [terrtia]
* [relationships] add messages mentions -> chats + user-accounts. [terrtia]
* [user-account] show chord diagram, Numbers of Messages Posted by Chat. [terrtia]
* [d3 heatmap] v7 migration + fix tooltip. [terrtia]
* [circos] add number of IN/OUT message in tooltip. [terrtia]
* [d3] get last D3 js version. [terrtia]
* [relationships] messages foraward between chats/user-account + chat chord diagram. [terrtia]
* [relationship] get object relationhips by relationship name + object types. [terrtia]
* [crawler] add function to delete schedules. [terrtia]
* [crawler stats] add previous month stats by domain type. [terrtia]
* [user message] show user messages by chat. [terrtia]
* [settings] refactor old settings blueprint module. [terrtia]
* [settings] refactor UI settings. [terrtia]
* [user session] refactor login + AIL users, use alternative IDs, kill user(s) session. [terrtia]
### Fix
* [crawler] fix crawler queue stats. [terrtia]
* [crawler] fix crawler queue stats. [terrtia]
* [crawler] fix crawler queue stats. [terrtia]
* [delete user] remove user from org. [terrtia]
* [crawler] crawler capture with empty task. [terrtia]
* [user] fix role edit. [terrtia]
* [user] user check. [terrtia]
* [object subtype dashboard] fix None first/last seen. [terrtia]
* [objects] get_object, None pgp name. [terrtia]
* [objects] get_object, filter None string. [terrtia]
* [trackers] typo. [terrtia]
* [trackers] blueprint user_role. [terrtia]
* [2fa] fix name. [terrtia]
* [org] fix trackers None orgs. [terrtia]
* [update v5.7] create roles list. [terrtia]
* [MISP Export] fix Decoded objects export. [terrtia]
* [user] fix users self password change. [terrtia]
* [users settings] fix user edit. [terrtia]
* [tracker] firce delete objects filter on edit. [terrtia]
* [showDomain] typo fixed. [Alexandre Dulaunoy]
* [debug] [terrtia]
* [user-account] fix chord graph tooltip. [terrtia]
* [trackers] fix webhook export. [terrtia]
* [dashboard] Filter out dashboard event-stream SSL.SSLEOFError from the log if a client disconnects. [terrtia]
* [trackers] fix tracker view objects acl for global trackers. [terrtia]
* [trackers] fix tracker view objects acl for global trackers. [terrtia]
* [trackers] fix post filters selector + fix tracker webhooks creation + fix set edit. [terrtia]
* [cookiejar] fix add cookie cookiejar uuid. [terrtia]
* [relationships] objects link. [terrtia]
* [user-account] fix last username timestamp. [terrtia]
### Other
* Merge branch 'master' into otp. [terrtia]
* Merge branch 'master' into otp. [terrtia]
* Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia]
* Update README.md. [Alexandre Dulaunoy]
* Merge branch 'relationships' into otp. [terrtia]
* Merge branch 'master' into relationships. [terrtia]
* Merge branch 'master' into otp. [terrtia]
## v5.6 (2024-05-10)
### Changes
* Chg: [tron] add TRON crytocurrency detection + correlation. Thanks @pventuzelo for the contribution. [terrtia]
* [tags] search ocrs and images by tags + fix ocr, filter invalid image. [terrtia]
* [correlation graph] add date first/last seen separator + fix display of ocr object tags. [terrtia]
### Fix
* [language detector] fix exception if the libretranslate url is not specified. [terrtia]
* [ocr] catch cv2 errors. [terrtia]
* [pgpdump] fix installer, new pgpdump version require to launch autoreconf. [terrtia]
* [ocr] catch OSError on MP4 files. [terrtia]
* [ocr] fix ocr supported languages. [terrtia]
* [Language] Updated the language detector to return a empty list when no language is detected. [terrtia]
* [ocr] filter ocr supported languages + fix type of object accepted by the tracker. [terrtia]
## v5.5 (2024-04-25)
### Changes
* [doc] add overview. [terrtia]
* [ocr] ocr daterange object, get ocrs by daterange + fixs. [terrtia]
* [reprocess tool] add OcrExtractor module + filter image gif. [terrtia]
* [ocr] add cache + correlation ocr-chats-messages + launch ocr extractor by default. [terrtia]
* [domains] card domain, fix last check. [terrtia]
* [ocr] detect and translate language + show ocr view + add languages blueprint. [terrtia]
* [ocr] get languages model + group extracted content by line + process ocr objects + get all images. [terrtia]
* [perf] reduce memory usage. [terrtia]
* [ocr] extract text from image + add ocr object. [terrtia]
* [tracker] add experimental report generator. [terrtia]
* [user-account] add heatmap nb user messages. [terrtia]
* [message] show trackers + modules matches. [terrtia]
* [trackers] UI: remove object from tracker. [terrtia]
* [user-account] show chats + subchannels. [terrtia]
### Fix
* [ocr] fix None copy date. [terrtia]
* [correlation] fix objects selector. [terrtia]
* [correlation] fix objects selector. [terrtia]
* [correlation] fix objects selector. [terrtia]
* [ocr] UI correlation, fix language bloc. [terrtia]
* [thehive] fix export logger. [terrtia]
* [UI matches extractor] fix empty matches. [terrtia]
* [UI matches extractor] handle overlapping matches. [terrtia]
* [languages] fix items language min probability. [terrtia]
### Other
* Merge branch 'master' into ocr. [terrtia]
* Merge branch 'master' into ocr. [terrtia]
## v5.4 (2024-03-25)
### Changes
* [language] add thread languages stats. [terrtia]
* [language messages] add nb languages stats by chat/subchannel objects. [terrtia]
* [lang] merge master. [terrtia]
* [lang] improve language detection + UI: manual translation and detection. [terrtia]
* [requirements] add lexilang. [terrtia]
* [trackers match] show objects date. [terrtia]
* [requirements] add demoji. [terrtia]
* [user-account] add default card. [terrtia]
* [chat-subchannel] add basic + default card. [terrtia]
* [chat] add chat default + basic card. [terrtia]
* [image object] add default + basic card. [terrtia]
* [global module] perf. [terrtia]
* [tools] reprocess objects by type. [terrtia]
* [modules] improve error output. [terrtia]
* [RetroHunt] retro hunt on messages. [terrtia]
* [correlation UI] change Direct Correlations position. [terrtia]
* [correlation] description: get message content + don't show etag, cookie_name and hhash by default. [terrtia]
* [chats] add heatmap nb messages/hour for all messages. [terrtia]
* [api] get chat/subchannel/thread messages. [terrtia]
* [api] get object + get investigation. [terrtia]
* [domains] add crawler status stats by domain type pie chart. [terrtia]
* [d3 js] add barchart_stack graph function. [terrtia]
* [crawler] add monthly crawled domains stats. [terrtia]
* [test] add basic API test. [terrtia]
* [git] update gitignore. [terrtia]
* [api] refactor blueprint. [terrtia]
* [favicon] add v5.4 update + update pylacus. [terrtia]
* [favicon] add favicons objects + correlation. [terrtia]
* [vanity] vanity tree, add links. [terrtia]
* [vanity] add vanity domain explorer + fix blurred screenshot + fix languages search filter. [terrtia]
* [domain] sort onion vanity name by prefix. [terrtia]
* [titles] add api to get unsafe titles. [terrtia]
* [titles] add api to get all titles. [terrtia]
* [tracker] list of last matchs, show description. [terrtia]
* [tracker] list of last matchs, show description. [terrtia]
* [retro_hunt] show match + fix None subtype. [terrtia]
* [titles] add endpoints to get all titles. [terrtia]
* [messages] manual translation by language + add new kvrocks lang namespace. [terrtia]
* [chat] add endpoints to download chat, subchannel and thread, + fix message translated by default. [terrtia]
* [doc] add screenshot. [terrtia]
* [exif] add debug. [terrtia]
* [doc] add discord/telegram chats JSON fields. [terrtia]
* [HOWTO] Libretranslate Chat translation. [terrtia]
* [requirement] bump flask requirement. [terrtia]
### Fix
* [chat] fix subchannel-message correlation + fix empty message language detection. [terrtia]
* [chat-subchannel] card empty chat. [terrtia]
* [dashboard] fix objects links. [terrtia]
* [tracker] fix yara content error on empty message. [terrtia]
* [FeederImporter] add all objects in queue. [terrtia]
* [module] fix SQLInjectionDetection object ID. [terrtia]
* [chats] fix templates, chat subtype. [terrtia]
* [api] remove debug output. [terrtia]
* [domains] add missing JS file. [terrtia]
* [crawler] fix monthly crawled domains stats svg size. [terrtia]
* [crawler] fix monthly crawled domains stats. [terrtia]
* [tests] fix tests + global new file content. [terrtia]
* [favicon] fix misp object export. [terrtia]
* [favicon] crawler favicon. [terrtia]
* [vanity] fix title. [terrtia]
* [domain] add screenshot img onerror. [terrtia]
* [vanity] length > 5 clusters. [terrtia]
* [vanity] debug. [terrtia]
* [vanity] debug. [terrtia]
* [vanity] length+1 min vanity = 2. [terrtia]
* [vanity] vanity builder. [terrtia]
* [vanity] update id domains is up + min vanity = 4. [terrtia]
* [api] get titles domain. [terrtia]
* [tracker] fix typo. [terrtia]
* [tracker] fix message trackers. [terrtia]
* [titles] add api to get all titles. [terrtia]
* [titles] add api to get all titles. [terrtia]
* [retro_hunt] api delete paused retro_hunt. [terrtia]
* [retro_hunt] delete paused retro_hunt. [terrtia]
* [doc] fix pystemon install. [terrtia]
* [doc] fix pystemon install. [terrtia]
* [flask] fix escape import. [terrtia]
* [chats] fix chats image importer. [terrtia]
* [language] catch libretranslate exception. [terrtia]
* [language] crawled items, force gcld3 detection. [terrtia]
* [LAUNCH] update screen status. [terrtia]
* [LAUNCH] fix killall. [terrtia]
### Other
* Merge lang. [terrtia]
* Merge branch 'master' of github.com:ail-project/ail-framework. [terrtia]
* Merge pull request #215 from NMD03/master. [Thirion Aurélien]
Add [ailbuilder] cleanup
* Add [ailbuilder] cleanup. [niclas]
## v5.3 (2024-02-05)
### Changes
* [LAUNCH] change restart flags. [terrtia]
* [UI] If version None do not show release note link. [Steve Clement]
* [LAUNCH] make sure reload works. [Steve Clement]
* [LAUNCH] Addded a "restart" option, killAll + launchAuto. [Steve Clement]
* [tags] add Tag class. [terrtia]
* [Hosts] improve perf + regex timeout + cache DNS results. [terrtia]
* [kvrocks] j -4 install + update to latest version. [terrtia]
* [translation] translate chats info, users info and subchannels names. [terrtia]
* [chats] show NB messages by chat. [terrtia]
* [v5.3] add v5.3 update. [terrtia]
* [relationships] add relationship engine + WIP relationships between forwarded messages/chats. [terrtia]
* [message] UI translate message object. [terrtia]
* [Language] replace pycld3 by gcld3 + clean text before language detection. [terrtia]
* [hosts] filter onion. [terrtia]
* [install] remove python package from thirdparty. [terrtia]
* [thirdparties] remove sb-admin + debug #194. [terrtia]
* [queues] timeout obj after 2 days. [terrtia]
* [chats] get user message ids by chat. [terrtia]
* [correlation] UI chats filters + correation user-account/username. [terrtia]
* [crawlers] add endpoints to reset captures. [terrtia]
* [crawlers] add endpoints to reset captures. [terrtia]
* [chats] add pagination. [terrtia]
* [chats] add participants corrlation shortcut. [terrtia]
* [tags] searech messages tags by daterange. [terrtia]
* [tags] search messages by tags. [terrtia]
* [categ] messages, bypass categ module + fix correlation. [terrtia]
* [crawler] push onion discovery capture_uuid to another AIL. [terrtia]
* [chats] add chats participants + user-accounts basic template. [terrtia]
* [chats] translate messages on demand. [terrtia]
* [chats] add discord threads, Forum channel. [terrtia]
* [chats] add messages threads. [terrtia]
* [chats] add message file-name object + str emoticon reactions. [terrtia]
* [chat users] add users icons. [terrtia]
* [chats] factorise heatmap + chat icon. [terrtia]
* [chats] add message and blur blocks. [terrtia]
* [queues] add new image queue + add exif module. [terrtia]
* [images] add sidebar shortcut. [terrtia]
* [chats] add image object + show message image. [terrtia]
* [chats] add heatmap nb week messages by hour. [terrtia]
* [chats] message object template. [terrtia]
* [submodules] bump. [terrtia]
* [messages] refactor get_messages_meta + add basic message template. [terrtia]
* [chats explorer] show chats/subchannels creation date. [terrtia]
* [chats] add UI shortcut + networks list + show chats/subchannels info. [terrtia]
* [chats] improve UI + fix importer. [terrtia]
* [chats] add chats explorer v0. [terrtia]
* [core] merge master + fix object subtype correlation stats. [terrtia]
* [crawlers] submit cookies to the crawler task API. [Jean-Louis Huynen]
* [crawlers] submit a single cookie to the crawler task API. [Jean-Louis Huynen]
* [mail exporter] add obj content extract for each regex match. [Terrtia]
* [mail exporter] add obj content extract for each yara rule match. [Terrtia]
* [titles] add yara tracker on title + tags domains if unsafe title tags. [Terrtia]
* [modules] crawl pasties domains. [Terrtia]
* [module + queues] track + rename object global ID by module. [terrtia]
* [importers obj_type] importers queues: add feeder source + object global ID. [terrtia]
* [ail queues] merge. [terrtia]
* [chat + user-account] correlations + usernames timeline. [Terrtia]
* [core] add telegram importer + Chat object + message Object + add timeline engine. [Terrtia]
* [module extrator] add debug. [Terrtia]
* [module extrator] add debug. [Terrtia]
* [telegram fedeer] use meta of the new feeder. [Terrtia]
* [flask] cleanup. [Terrtia]
* [flask] cleanup, remove unused import. [Terrtia]
* [crawler ui] last crawled domains, show last check timestamp. [Terrtia]
* [tracker ui] improve show typo squatting button + add tooltip. [Terrtia]
* [api crawler] fix response + add cookiejar, proxy and frequency parameters. [Terrtia]
* [HHHash] add HHHash object and correlation https://www.foo.be/2023/07/HTTP-Headers-Hashing_HHHash. [Terrtia]
* [objs processed] xxhash messages. [terrtia]
* [queues] track object + check if object processed. [Terrtia]
### Fix
* [LAUNCH] fix ENV error message. [terrtia]
* [crawler] fix capture None domain name. [terrtia]
* [tags] fix invalid tags. [terrtia]
* [README] api.md link typo. [fukusuket]
* [tags] fix typo. [terrtia]
* [docs + UI] fix repository links. [terrtia]
* [ui] Do not show relNotes link if not on a tag. [Steve Clement]
* [repo] Point to the correct ail-project repo. [Steve Clement]
* [D4] fix module cache. [terrtia]
* [misp export] fix empty event on module start. [terrtia]
* [crawlers] fix capture return error code. [terrtia]
* [crawlers] fix errored capture queue. [terrtia]
* [crawlers] fix errored capture start time. [terrtia]
* [chat view] fix created_at + filter. [terrtia]