Research the detection of the primary name for an identified person

[src-d]

Research the detection of the primary name for an identified person

[vmarkovtsev]

The primary user names are included in GHTorrent.
@EgorBu Can you please attach here the best identity matching results on GitHub?

[vmarkovtsev]

I collected the dataset of full names: /user/internal-datasets/full_names

Random strategy

46% accuracy if we consider identities without a correct name. Otherwise, ~82%.

Biggest number of commits strategy

56% and 99.9%, respectively.

So 99.9% is a very good number and there is no need for digging deeper. We should count commits for each name and sort them, problem solved.

[EgorBu]

I will launch the extraction of identities and dump it here

Update:
Old result found here: /user/egorbu/idmatching/cache/res_name_threshold_5_email_threshold_28_data_aggregated_deduplicated.pkl - it was launched on initial dataset (not full)
I will launch extraction on full dataset.

[vmarkovtsev]

No, the user names are no longer available. I can take all the logins from GHTorrent, but the names require some GitHub crawling. I will have to run it.

[vmarkovtsev]

I am crawling the full name for each github login on our 11 nodes. ETA 10 days.

[vmarkovtsev]

Fcking GitHub is throttling us with 429. I had to add download delays using the Scrapy's AutoThrottle extension, that will most probably increase ETA.

#38

Fetched ~900k in 10 hours.

[vmarkovtsev]

My experiments showed that the rate-limiting happens if you do more than or equal to 5 requests per second. I set the frequency to 4 and the process has finally stabilized.

[vmarkovtsev]

Progress: 4.6mm through the weekend