Developed from the ground-up a new analysis framework for experiments at Udemy, incorporating:
Variance reduction methods.
Sample sizing methods that adjust for "peeking" as well as methods to choose the "optimal" level of peeking from the perspective of expected experiment duration.
Novel methods to reduce variance in quantile treatment effects.
Developed a pre-experiment tool for metric selection based on composing multiple metrics into an ideal blended metric.
Developed an estimator that is robust to "sample ratio mismatch" issues, allowing the system to use a correct-not-test approach.
Wrote educational material on how to make decisions about experiments, interpret results as well as did short video tutorials.
Developed an improved pricing model that:
Unified structural econometric techniques and causal inference methods into a unique pricing approach.
Modeled substitution between various products and product types.
Delivered pricing wins from fine-tuning the model for different markets.
Developed a long-run, downstream impact model for experiments to give a bottom-line measured of impact across different product types (transactional and subscription purchases).
Assisted economic experts testifying on a variety of matters in antitrust (mergers, price-fixing, output restriction) and financial cases (allegations of "spoofing", alleged unlawful securities) by providing data analysis and economic analysis.
Research Scientist, Research and Development, Washington, DC.
July 1, 2018 to January 2020.
Applied and developed models and built the tools to operationalize those models. A hybrid research scientist/engineer role.
The company builds models to decide how best to pair callers and call center agents. It bills based on how much it increases, say, the sales rate or reduces the cancellation rate.
I developed
an internal R package and strategy to predict the performance of models in production before they are deployed.
models that determined which call center agents and callers should be paired to maximize our client’s objectives.
a main component of the application we use to decide how to route calls between callers and agents (a production application in C++).
metrics to monitor how well different strategies for pairing callers and agents do in production once they are deployed.
research on new, next-generation matching algorithms.
Designed, developed, and used a discrete choice demand model to help make pricing and product design decisions that was used in a wide variety of applications at the company. For example, it was used
to inform major Prime pricing decisions;
to separately value the various benefits of being a Prime member (free shipping, streaming video and music, etc)
to estimate the effect of modifying Prime benefits on Prime membership;
to estimate the effect of potential video content investment decisions on Prime membership.
Developed a censored quantile regression approach to model what factors make customers buy a product or sign up for Prime sooner.
Worked on many smaller projects, including analysis of which cities had the greatest opportunity size for advertising Prime during NFL games and which lower-income markets had the greatest opportunity size for advertising a lower-priced Prime for lower-income customers.
Implemented dynamic discrete choice models to understand housing decision problems for two projects: one on post-Katrina New Orleans and the other on the national housing market.
Used Condor at the CHTC at the University of Wisconsin Madison to estimate the large scale dynamic discrete choice models.
Teaching Assistant
Introduction to Macroeconomics (Spring 2013, Spring 2014), Economic Statistics (Fall 2013), Introduction to Microeconomics (Fall 2012)
Programming languages and computing
Environments and programs: AWS, Linux, Databricks, Hive, Spark.
For some examples of projects I have done/jobs where I used the above languages regularly:
C/C++ — I used C++ for building on the core product at Afiniti and I have written C code for personal projects (see Github).
R — All of my academic papers were mostly written in R (and Fortran). The censored quantile regression project at Amazon mentioned above was done in R as well. I used it frequently for projects at Compass Lexecon and Afiniti, including developing a widely-used internal R package for validating models.
Python — I used Python for various internal projects at Afiniti, a few cases at Compass Lexecon, and for essentially all projects at Udemy, including the experiment analysis system mentioned above.
Julia — I worked in Julia for an internal project at Afiniti.
I used various SQL flavors daily at Afiniti, Amazon, and Udemy.
Stata — Some of my RA work at Wisconsin was in Stata, the paper with Lenadro Magnusson developed a Stata package, and I used Stata at Amazon and Compass Lexecon for many projects.
RA work with Jesse Gregory used both Matlab and Fortran.
I use Perl in personal scripts, and I used it to automate certain tasks at Amazon.
I used SAS in cases at Compass Lexecon.
Writing
I write a blog on economics and data science issues at https://zachflynn.substack.com, and I've written academic papers in economics.