penny4860/svhn-deep-digit-detector

Deep-digit-detector (and recognizer) in natural scene. A digit detection framework was implemented using keras with tensorflow backend.

svhn-deep-digit-detector's Issues

hard negative mining

package setup

setup.py 를 이용해서 package 를 setup 하자.

refactoring recognizer

No module named 'crop'

Trying to start the process of getting this digit detector up and running. Have not gotten very far. Attempting to run 1_sample_loader.py is dependent on extractor.py which imports region_proposal.py.

In region_proposal.py it imports 'crop' and 'show', where are these coming from. There are no modules with these names in "A list of all the packages needed to run this project can be found in digit_detector.yml."

non-maximum suppression

accurate region proposals

• Threshold images by a set of [0, 255]
• connected-component-labeling

hog-logisticreg-detector

descriptor : Histogram of Gradient
classifier : logistic regression

code refactoring

arranging channel order in show module

positive samples

• 현재 인식이 정확하지 않다.
• region proposal 중에서 overlap 이 75% 이상이면 positive sample 에 추가해보자.
• ground truth 에서 margin (padding) 을 조금씩 주자.

descriptor refactoring

descriptor 에서

• mean_value subtraction
• (N, n_rows, n_cols, 1) 로 reshape

MSER

请问MSER算法是作用在哪一环节的呢？

negative sample 수집할 때 padding 을 반영

candidate region 을 classifier 에 입력할 때와 같은 방식으로 negative sample 을 수집해보자.

train detector

• 32x32x1
• Positive Samples
  • svhn matlab file load
• Negative Samples
  • random cropping in natural scene
    • 1. remove digit region in SVHN natural scene
    • 2. random crop

Format annotation file

I tried to load the data thanks to "1_sample_loader.py" which requires a digitStruct.json file. Inside the train.tar.gz file there is only a digitStruct.mat file. Have you done any manipulation to convert it in matlab from .mat to .json ?

the current state of the art in objects classification

• http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html#5356484e 에서 svhn 데이터셋 학습시킨 논문들 참조하기.

run script test

extract features

Run script for the detector

• Run the classifier to the candidate regions
• bounding-box 를 original image 에 표시

data augmentation

• Translation
• Rotation
  • -15 ~ +15 degree
• Noise

https://keras.io/preprocessing/image/ 라이브러리 사용하기
https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html
http://pastebin.com/0QHtPGzJ

• positive sample : negative sample 의 비율이 1:100 이다. positive sample 에 대해서만 data augmentation 해서 sample 간 balance 를 맞춰보자.
  • https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html 참조
  • ImageDataGenerator.flow(X, y) 로 X_aug 를 generate 해서 저장
  • (X, X_aug) 를 합쳐서 data 숫자를 늘려보자.

evaluate detector's performance

• Performance of the region proposer

  • recall value : 0.630,
  • precision value : 0.045
  • f1_score : 0.084

• Performance of the detector

  • recall value : 0.487
  • precision value : 0.656
  • f1_score : 0.559

pruning

region proposal 중에서 가로의 길이가 더 긴 것을 pruning

pytest unittest setup

Improving Idea

• 초기화
  • 학습된 network 로 transfer learning
• Validation Data 를 original sample 에서만 sampling
• Bounding Box 에 Margin 을 줘서 crop 하자
  • 32-32 size 의 mat file을 그대로 사용하는 방법
  • Bounding Box 에서 ratio별로 margin 을 주는 방법
• Data Augmentation 을 Negative Sample 도 같이 하자
  • Training 할 때 run-time 으로
• Hard-Negative Sample 을 더 추가

Pickle

The trained model and the get_preds function work perfectly fine but when I export the trained model as a pickle file and then try to use it, it doesn't work. Maybe it is an issue of fastai. When I updated fastai to the latest version, no one of the previous functions run. Does anyone have a pkl, hdf5 or any other trained model for use?

Recognizer Training

extractor module refactoring

CNN-descriptor

evaluation script

1개의 image 에서 여러개의 ground truth 가 있을 때 mAP 를 구할 수 있도록 코드를 수정하자.

• multiple object 에 대한 evaluation 구현
• Test data 에 대한 mAP 를 구할 수 있도록 setup

Resizing Candidate Proposals

학습된 모델 (32x32x1 => digit or not) 을 natural image 에서 돌리는 코드 구현

• 입 출력
  • Input : Natural Image
  • Output : Candidate Regions whose shape is (N, 32, 32, 1)
• 구현할 내용
  • 입력 영상을 Gray Scale 로 변환
  • MSER 로 Candidate Region 을 찾는다
  • Candidate Region 을 32x32x1 로 resize
    • (w >= h) : 32x32 로 rescale
    • (w < h) : w=h 가 되도록 crop 후 32x32 로 rescale
      • natural 영상의 edge 부에서의 처리 ?

Factory of Detector implementation

Detector 를 생성하는 factory method를 구현

hard-negative-mine

imshow() bug

In 5_run.py if there are more than 2-test-images, the 2nd image cannot display.

the error message is as follows

init done 
opengl support available 

Muti-Task Learning

Muti-Task Learning

• digit detector
• digit recognizer