Skip to content

Commit d8fcff6

Browse files
bnookalabnookala
authored
Rebase upstream repository (#100)
1 parent 18976b8 commit d8fcff6

File tree

56 files changed

+1678
-293
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+1678
-293
lines changed

Diff for: ‎README.md

+8-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# Active learning + object detection
22
Labeling images for object detection is commonly required task to get started with Computer Vision related project.
3-
Good news that you do not have to label all images (draw bounding boxes) from scratch --- the goal of this project is to add (semi)automation to the process.
3+
Good news that you do not have to label all images (draw bounding boxes) from scratch --- the goal of this project is to add (semi)automation to the process.
4+
Please refer to this blog post that describes Active Learning and semi-automated flow:
5+
[Active Learning for Object Detection in Partnership with Conservation Metrics](https://www.microsoft.com/developerblog/2018/11/06/active-learning-for-object-detection/)
46
We will use Transfer Learning and Active Learning as core Machine Learning components of the pipeline.
57
-- Transfer Learning: use powerful pre-trained on big dataset (COCO) model as a startining point for fine-tuning foe needed classes.
68
-- Active Learning: human annotator labels small set of images (set1), trains Object Detection Model (model1) on this set1 and then uses model1 to predict bounding boxes on images (thus pre-labeling those). Human annotator reviews mode1's predictions where the model was less confident -- and thus comes up with new set of images -- set2. Next phase will be to train more powerful model2 on bigger train set that includes set1 and set2 and use model2 prediction results as draft of labeled set3…
@@ -28,6 +30,11 @@ There is config.ini that needs to be updated with details like blob storage conn
2830
More details TBD.
2931
Basically the idea is to kick off Active Learning cycle with model retaining as soon as human annotator revises new set of images.
3032

33+
# Notes before we get started
34+
- The steps below refer to updating config.ini. You can find detailed description of config [here](config_description.md)
35+
- Got several thousands of images (or much more) and not sure if random sampling will be helpful to get rolling with labeling data?
36+
Take a look at [Guide to "initialization" predictions](init_pred_desription.md).
37+
3138
# How to run semi-automated pipeline
3239
The flow below assumes the following:
3340
1) We use Tensorflow Object Detection API (Faster RCNN with Resnet 50 as default option) to fine tune object detection.

Diff for: ‎config.ini

+6-1
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ min_confidence=.5
3939
test_percentage=.2
4040
model_name=faster_rcnn_resnet50_coco_2018_01_28
4141
optional_pipeline_url=https://raw.githubusercontent.com/tensorflow/models/master/research/object_detection/samples/configs/faster_rcnn_resnet50_pets.config
42+
#Init Predictions
43+
init_model_name=faster_rcnn_resnet101_coco_2018_01_28
4244
# Config File Details
4345
old_label_path=PATH_TO_BE_CONFIGURED/pet_label_map.pbtxt
4446
old_train_path=PATH_TO_BE_CONFIGURED/pet_faces_train.record-?????-of-00010
@@ -65,4 +67,7 @@ tf_val_record=${tf_record_location%.*}_val.${tf_record_location##*.}
6567
tf_url=http://download.tensorflow.org/models/object_detection/${model_name}.tar.gz
6668
pipeline_file=${download_location}/${model_name}/pipeline.config
6769
fine_tune_checkpoint=${download_location}/${model_name}/model.ckpt
68-
tagging_output=${data_dir}/tagging.csv
70+
tagging_output=${data_dir}/tagging.csv
71+
init_pred_tf_url=http://download.tensorflow.org/models/object_detection/${init_model_name}.tar.gz
72+
init_model_graph=${download_location}/${init_model_name}/frozen_inference_graph.pb
73+

Diff for: ‎functions/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -208,4 +208,4 @@ Showing our function running:
208208
```bash
209209
curl "https://jmsactlrnpipeline.azurewebsites.net/api/download?code=AARPr45D5K6AIEWv8bEaqWalSaddrUzd4aydOxmhSPauGUrsPvzw==&imageCount=1"
210210
["https://csehackstorage.blob.core.windows.net/image-to-tag/1.jpg"]
211-
```
211+
```

Diff for: ‎functions/pipeline/labels/__init__.py

+1
Original file line numberDiff line numberDiff line change
@@ -117,4 +117,5 @@ def __create_ImageTag_list(image_id, tags_list):
117117
image_tags = []
118118
for tag in tags_list:
119119
image_tags.append(ImageTag(image_id, tag['x1'], tag['x2'], tag['y1'], tag['y2'], tag['classes']))
120+
120121
return image_tags

Diff for: ‎functions/pipeline/onboardcontainer/__init__.py

+111-111
Original file line numberDiff line numberDiff line change
@@ -1,111 +1,111 @@
1-
import os
2-
import logging
3-
import json
4-
import azure.functions as func
5-
from urlpath import URL
6-
from datetime import datetime, timedelta
7-
from ..shared.constants import ImageFileType
8-
from ..shared.storage_utils import get_filepath_from_url
9-
10-
from azure.storage.blob import BlockBlobService, BlobPermissions
11-
from azure.storage.queue import QueueService, QueueMessageFormat
12-
13-
DEFAULT_RETURN_HEADER = {
14-
"content-type": "application/json"
15-
}
16-
17-
18-
def main(req: func.HttpRequest) -> func.HttpResponse:
19-
logging.info('Python HTTP trigger function processed a request.')
20-
21-
user_name = req.params.get('userName')
22-
23-
if not user_name:
24-
return func.HttpResponse(
25-
status_code=401,
26-
headers=DEFAULT_RETURN_HEADER,
27-
body=json.dumps({"error": "invalid userName given or omitted"})
28-
)
29-
30-
try:
31-
req_body = req.get_json()
32-
logging.debug(req.get_json())
33-
storage_account = req_body["storageAccount"]
34-
storage_account_key = req_body["storageAccountKey"]
35-
storage_container = req_body["storageContainer"]
36-
except ValueError:
37-
return func.HttpResponse(
38-
"ERROR: Unable to decode POST body",
39-
status_code=400
40-
)
41-
42-
if not storage_container or not storage_account or not storage_account_key:
43-
return func.HttpResponse(
44-
"ERROR: storage container/account/key/queue not specified.",
45-
status_code=401
46-
)
47-
48-
# Create blob service for storage account (retrieval source)
49-
blob_service = BlockBlobService(
50-
account_name=storage_account,
51-
account_key=storage_account_key)
52-
53-
# Queue service for perm storage and queue
54-
queue_service = QueueService(
55-
account_name=os.getenv('STORAGE_ACCOUNT_NAME'),
56-
account_key=os.getenv('STORAGE_ACCOUNT_KEY')
57-
)
58-
59-
queue_service.encode_function = QueueMessageFormat.text_base64encode
60-
61-
try:
62-
blob_list = []
63-
64-
for blob_object in blob_service.list_blobs(storage_container):
65-
blob_url = URL(
66-
blob_service.make_blob_url(
67-
storage_container,
68-
blob_object.name
69-
)
70-
)
71-
# Check for supported image types here.
72-
if ImageFileType.is_supported_filetype(blob_url.suffix):
73-
logging.debug("INFO: Building sas token for blob " + blob_object.name)
74-
# create sas signature
75-
sas_signature = blob_service.generate_blob_shared_access_signature(
76-
storage_container,
77-
blob_object.name,
78-
BlobPermissions.READ,
79-
datetime.utcnow() + timedelta(hours=1)
80-
)
81-
82-
logging.debug("INFO: have sas signature {}".format(sas_signature))
83-
84-
signed_url = blob_url.with_query(sas_signature)
85-
86-
blob_list.append(signed_url.as_uri())
87-
88-
logging.debug("INFO: Built signed url: {}".format(signed_url))
89-
90-
msg_body = {
91-
"imageUrl": signed_url.as_uri(),
92-
"fileName": str(blob_url.name),
93-
"fileExtension": str(blob_url.suffix),
94-
"directoryComponents": get_filepath_from_url(blob_url, storage_container),
95-
"userName": user_name
96-
}
97-
98-
body_str = json.dumps(msg_body)
99-
queue_service.put_message("onboardqueue", body_str)
100-
else:
101-
logging.info("Blob object not supported. Object URL={}".format(blob_url.as_uri))
102-
103-
return func.HttpResponse(
104-
status_code=202,
105-
headers=DEFAULT_RETURN_HEADER,
106-
body=json.dumps(blob_list)
107-
)
108-
except Exception as e:
109-
logging.error("ERROR: Could not build blob object list. Exception: " + str(e))
110-
return func.HttpResponse("ERROR: Could not get list of blobs in storage_container={0}. Exception={1}".format(
111-
storage_container, e), status_code=500)
1+
import os
2+
import logging
3+
import json
4+
import azure.functions as func
5+
from urlpath import URL
6+
from datetime import datetime, timedelta
7+
from ..shared.constants import ImageFileType
8+
from ..shared.storage_utils import get_filepath_from_url
9+
10+
from azure.storage.blob import BlockBlobService, BlobPermissions
11+
from azure.storage.queue import QueueService, QueueMessageFormat
12+
13+
DEFAULT_RETURN_HEADER = {
14+
"content-type": "application/json"
15+
}
16+
17+
18+
def main(req: func.HttpRequest) -> func.HttpResponse:
19+
logging.info('Python HTTP trigger function processed a request.')
20+
21+
user_name = req.params.get('userName')
22+
23+
if not user_name:
24+
return func.HttpResponse(
25+
status_code=401,
26+
headers=DEFAULT_RETURN_HEADER,
27+
body=json.dumps({"error": "invalid userName given or omitted"})
28+
)
29+
30+
try:
31+
req_body = req.get_json()
32+
logging.debug(req.get_json())
33+
storage_account = req_body["storageAccount"]
34+
storage_account_key = req_body["storageAccountKey"]
35+
storage_container = req_body["storageContainer"]
36+
except ValueError:
37+
return func.HttpResponse(
38+
"ERROR: Unable to decode POST body",
39+
status_code=400
40+
)
41+
42+
if not storage_container or not storage_account or not storage_account_key:
43+
return func.HttpResponse(
44+
"ERROR: storage container/account/key/queue not specified.",
45+
status_code=401
46+
)
47+
48+
# Create blob service for storage account (retrieval source)
49+
blob_service = BlockBlobService(
50+
account_name=storage_account,
51+
account_key=storage_account_key)
52+
53+
# Queue service for perm storage and queue
54+
queue_service = QueueService(
55+
account_name=os.getenv('STORAGE_ACCOUNT_NAME'),
56+
account_key=os.getenv('STORAGE_ACCOUNT_KEY')
57+
)
58+
59+
queue_service.encode_function = QueueMessageFormat.text_base64encode
60+
61+
try:
62+
blob_list = []
63+
64+
for blob_object in blob_service.list_blobs(storage_container):
65+
blob_url = URL(
66+
blob_service.make_blob_url(
67+
storage_container,
68+
blob_object.name
69+
)
70+
)
71+
# Check for supported image types here.
72+
if ImageFileType.is_supported_filetype(blob_url.suffix):
73+
logging.debug("INFO: Building sas token for blob " + blob_object.name)
74+
# create sas signature
75+
sas_signature = blob_service.generate_blob_shared_access_signature(
76+
storage_container,
77+
blob_object.name,
78+
BlobPermissions.READ,
79+
datetime.utcnow() + timedelta(hours=1)
80+
)
81+
82+
logging.debug("INFO: have sas signature {}".format(sas_signature))
83+
84+
signed_url = blob_url.with_query(sas_signature)
85+
86+
blob_list.append(signed_url.as_uri())
87+
88+
logging.debug("INFO: Built signed url: {}".format(signed_url))
89+
90+
msg_body = {
91+
"imageUrl": signed_url.as_uri(),
92+
"fileName": str(blob_url.name),
93+
"fileExtension": str(blob_url.suffix),
94+
"directoryComponents": get_filepath_from_url(blob_url, storage_container),
95+
"userName": user_name
96+
}
97+
98+
body_str = json.dumps(msg_body)
99+
queue_service.put_message("onboardqueue", body_str)
100+
else:
101+
logging.info("Blob object not supported. Object URL={}".format(blob_url.as_uri))
102+
103+
return func.HttpResponse(
104+
status_code=202,
105+
headers=DEFAULT_RETURN_HEADER,
106+
body=json.dumps(blob_list)
107+
)
108+
except Exception as e:
109+
logging.error("ERROR: Could not build blob object list. Exception: " + str(e))
110+
return func.HttpResponse("ERROR: Could not get list of blobs in storage_container={0}. Exception={1}".format(
111+
storage_container, e), status_code=500)

Diff for: ‎images/VOTT_animal.PNG

2.06 MB
Loading

Diff for: ‎images/init_predict.PNG

60.3 KB
Loading

Diff for: ‎init_pred_desription.md

+76
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Guide to "initialization" predictions
2+
Assuming you got an datatet containing many thousands of images -- how do you get started with labeling first
3+
few hundreds images?
4+
What about unblanced case when most of the pictures not have much going on?
5+
If you just random sample pictures _blindly_ it make quite a few Active Learning cycles to set your model and
6+
training set onto the right pass.
7+
8+
## Let's get "metadata" about each image
9+
We could use pretrained model that can detect decently few dozens or more object class to get idea what kind
10+
of objects are on the images. The model might not provide super-accurate results however some of those might be
11+
useful for more target image sampling.
12+
For example if you dataset has common scenes of nature or city life than using model trained on [COCO dataset](https://github.com/amikelive/coco-labels/blob/master/coco-labels-paper.txt)
13+
might give you an idea what images have objects that _resembles_ person, car, deer and so on. And depedning on your
14+
scenario you might focus you initial labeling efforts on images that have or don't have a particular class.
15+
16+
![Flow](images/init_predict.PNG)
17+
18+
## Settings in config.ini
19+
The following settings control what model is going to be used for "initialization" predictions.
20+
- init_model_name=faster_rcnn_resnet101_coco_2018_01_28
21+
Model name to be used for predictions. Current code assumes it's COCO based model.
22+
- init_pred_tf_url=http://download.tensorflow.org/models/object_detection/${init_model_name}.tar.gz
23+
URL for downloading model from Tensorflow Object Detection model zoo.
24+
- init_model_graph=${download_location}/${init_model_name}/frozen_inference_graph.pb
25+
Location (on DSVM) of inference graph that's used for producing "initialization" predictions.
26+
27+
## Running "initialization" predictions flow
28+
Once config settings are set (and images are on blob storage) user needs to do the following:
29+
- SSH to DSVM and run script that will actually produces predictions
30+
- provide desired mapping (and merging) or detected classes to the classes of interest (more details below)
31+
- dowload specified number of images to client machine and review the tags
32+
33+
*Produce predictions*
34+
SSH to DSVM, activate needed Tensorflow virtul environment if needed and run:
35+
`. ./active_learning_init_pred.sh ../config.ini`
36+
The output _init_totag*.csv_ contains all detecting objetcs bboxes iformation. It's probably worth spedining
37+
the time analizing those results.
38+
39+
*Class mapping json*
40+
Please refer to _sample_init_classes_map.json_ for reference.
41+
First we define that we want to class "1" to be shown as class "person" in VOTT when user will be doing labels review.
42+
We also want to have 60% of images that will be pulled for review to have presence of class "person" in them:
43+
`{`
44+
`"initclass": "1", `
45+
`"map": "person",`
46+
`"balance": "0.6"`
47+
`}`
48+
49+
Then we want to _merge_ several classes: "19" (horse) and "21" (cow) will be displayed in VOTT as "animal".
50+
`{`
51+
`"initclass": "19",`
52+
`"map": "animal",`
53+
`"balance": "0.2"`
54+
`},`
55+
`{`
56+
`"initclass": "21",`
57+
`"map": "animal",`
58+
`"balance": "0.2"`
59+
`}'
60+
61+
We specify that 20% of each _animal_ class (40% in total) is present in the dataset that user will be reviewing in VOTT.
62+
Also we specifically request not to include images where no known COCO classes were detected. Given that COCO-based
63+
model may miss quite a bit of objects it's good practice still to review some of those.
64+
Model might be detecting classes that we will be cluttering image during review process. For example the dataset
65+
may have busket images that is wrongly classified as a "vase". In scenario when we are not interested in detecting
66+
baskets nor vases we may want just to "drop" bboxes for the "vase" class (class 86 in COCO):
67+
` "unmapclass":["64","86", "15"],`
68+
Finally for _everything else_ -- classes we are not sure what to do at this stage but still want to preserve bbox --
69+
we will map then to a "default" class. We can set the name of "default" class in mapping json.
70+
71+
*Review predictions in VOTT*
72+
On a client (tagger) machine run the usual script to download images. Only difference is that you'd be providing
73+
"class mapping json" as 3rd parameter:
74+
` D:\repo\active-learning-detect\tag>python download_vott_json.py 200 ..\config.ini ..\sample_init_classes_map.json`
75+
76+
![Flow](images/VOTT_animal.PNG)

0 commit comments

Comments
 (0)