microsoft/longitudinaldifferenceindifferencespy

Did my program cause the improvement in my numbers?

Introduction

It's critical to know if the work we are doing is helping the team reach its organizational goals. The problem is that there are potentially many other effects at play. One analysis method for doing this is called a difference in differences analysis. If I had done nothing, then what would have happened to the numbers I care about or "key performance indicators" (KPI) after people started using my product.

It's not always feasible (e.g. due to cost or lack of necessary infrastructure) or ethical (e.g. randomly assigning a cancer-causing drug to a subgroup of the population sample) to run a fully randomized experiment when building a product. If you're looking to measure impact on some quantitative KPIs and you can't run experiments/"randomized control trials"/"AB Testing", then this type of analysis is for you.

Thanks to Sol Sadeghi (Cosine Data Science team @ Microsoft) for informing me about this analysis technique and helping with the initial code review!

Case Study

This package is designed for studying the causal effect of WAVE using code authoring time – result of a research study done by MSR and Windows ES, on developers’ productivity. This led the WAVE team deciding on automatically installing their product to all relevant COSINE developers.

Example of how this tool was used inside of Microsoft to show the (5+%) impact of a product on internal/"1st party" developer productivity within the Windows division at Microsoft coming soon.

Getting Started

Our documentation is available on this repository's wiki.

Demo

To see a full demonstration of the tool with some example data,

1. Navigate to the example directory.
2. Run the following command:

python exampleDIDAnalysis.py

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

longitudinaldifferenceindifferencespy's Issues

Open source WAVE decision document!

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Support other ways to compare to comparison group

Today, the comparison groups metrics all come from the same point in time. We should support other ways of calculating the metrics with the individual data from different points in time, since we have varied time points in the treatment group.

• Matching
• Propensity score matching

Move wiki documentation over here

Add support to output the pre and post values that are graphed for treatment and comparison groups

When trying to understand the percentage improvement of a treatment, getting the outputs of the pre and post values of treatment and comparison would be super helpful. Bigger plus would be to have the treatment % impact outputted along with the set of outputs already printed to the CMD today.

No screenshot for statistical estimates

Add library to PyPI