k-threshold-impact-on-reporting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  k identical reports threshold impact on reporting "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prologue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below, you will find a form of pseudo code we used to pull the data underlying the blog post Criteo published about k-thresholding.\n",
    "\n",
    "The raw data we base our analysis upon is stored in one Hadoop table (`logs.displays`) with the following structure:\n",
    "\n",
    "| field name | data type | description |\n",
    "|------------|---------- |-------------|\n",
    "| day | DATE | Date at which the displays was printed |\n",
    "| environment | STRING | environment of the display (web, app, etc.) |\n",
    "| advertiser | STRING | Advertiser name (representing the interest group in this context) |\n",
    "| publisher | STRING | Publisher name |\n",
    "| displays | BIGINT | Number of displays |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import querying_package\n",
    "import os\n",
    "import pandas as pd\n",
    "import math\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Publisher Side Reporting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pseudo query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = f\"\"\"\n",
    "    SELECT\n",
    "        d.advertiser AS advertiser\n",
    "        , d.publisher AS publisher\n",
    "        , SUM(d.displays)/(1 + day(date '2020-06-17' - date '2020-06-11')) as nb_display\n",
    "    FROM\n",
    "        logs.displays d\n",
    "    WHERE\n",
    "        d.day BETWEEN '2020-06-11' AND '2020-06-17'\n",
    "        AND d.environment = 'web'\n",
    "        AND d.publisher IN ('my_major_international_publisher', 'my_medium_publisher', 'my_small_publisher', 'my_extra_small_publisher')\n",
    "    GROUP BY\n",
    "        d.advertiser\n",
    "        , d.publisher\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We pull the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = \"./data/20200630-publisher-side-obfuscated.csv\"\n",
    "\n",
    "csv_exists = os.path.isfile(data_path)\n",
    "if csv_exists:\n",
    "    df = pd.read_csv(data_path) \n",
    "else:\n",
    "    df = querying_package.run_presto_query(query, output=data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the log scale for better visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['nb_display'] = df['nb_display'].astype('float')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Share of unreported advertisers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aggregate the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp = df.groupby(['publisher', 'nb_display'])[['advertiser']].agg('count').sort_index()\n",
    "tmp['n_advertisers_cum_sum'] = tmp.groupby(level=['publisher']).cumsum()\n",
    "tmp['n_advertisers_running_pct'] = tmp.groupby(level=['publisher'])['n_advertisers_cum_sum'].transform(lambda x: x / x.iloc[-1])\n",
    "tmp = tmp.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(figsize=(18, 9))\n",
    "ax.grid(False)\n",
    "sns.lineplot(data=tmp\n",
    "             , x='nb_display'\n",
    "             , y='n_advertisers_running_pct'\n",
    "             , hue='publisher', hue_order = ['Major International Publisher', 'Medium Publisher', 'Small Publisher', 'Extra-Small Publisher']\n",
    "             , linewidth=3)\n",
    "plt.xlabel('k threshold', fontsize=16)\n",
    "plt.xscale('log')\n",
    "plt.ylabel('Share of unreported advertisers', fontsize=16)\n",
    "ax.set(ylim=(-0.1, 1.1))\n",
    "plt.title('Share of unreported advertisers in daily reporting with k-thresholding', fontsize=20)\n",
    "plt.gca().yaxis.set_major_formatter(ticker.PercentFormatter(xmax=1))\n",
    "\n",
    "handles, labels = ax.get_legend_handles_labels()\n",
    "ax.legend(handles=handles[1:], labels=labels[1:], prop={'size': 16})\n",
    "\n",
    "plt.axvline(9, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(9+0.5, 0.8, f\"k = 10\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(99, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(99+5, 0.8, f\"k = 100\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(999, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(999+50, 0.8, f\"k = 1000\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(9999, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(9999+500, 0.8, f\"k = 10000\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.savefig(f'./output/20200630-publisher-report-nadvertisers.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Share of unreported displays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aggregate the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['nb_display_dim'] = df['nb_display']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp = df.groupby(['publisher', 'nb_display_dim'])[['nb_display']].agg('sum').sort_index()\n",
    "tmp['n_displays_cum_sum'] = tmp.groupby(level=['publisher']).cumsum()\n",
    "tmp['n_displays_running_pct'] = tmp.groupby(level=['publisher'])['n_displays_cum_sum'].transform(lambda x: x / x.iloc[-1])\n",
    "tmp = tmp.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(figsize=(18, 9))\n",
    "ax.grid(False)\n",
    "sns.lineplot(data=tmp\n",
    "             , x='nb_display_dim'\n",
    "             , y='n_displays_running_pct'\n",
    "             , hue='publisher', hue_order = ['Major International Publisher', 'Medium Publisher', 'Small Publisher', 'Extra-Small Publisher']\n",
    "             , linewidth=3)\n",
    "plt.xlabel('k threshold', fontsize=16)\n",
    "plt.xscale('log')\n",
    "plt.ylabel('Share of unreported displays', fontsize=16)\n",
    "ax.set(ylim=(-0.1, 1.1))\n",
    "plt.title('Share of unreported displays in daily reporting with k-thresholding', fontsize=20)\n",
    "plt.gca().yaxis.set_major_formatter(ticker.PercentFormatter(xmax=1))\n",
    "\n",
    "handles, labels = ax.get_legend_handles_labels()\n",
    "ax.legend(handles=handles[1:], labels=labels[1:], prop={'size': 16})\n",
    "\n",
    "plt.axvline(9, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(9+0.5, 0.8, f\"k = 10\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(99, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(99+5, 0.8, f\"k = 100\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(999, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(999+50, 0.8, f\"k = 1000\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(9999, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(9999+500, 0.8, f\"k = 10000\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.savefig(f'./output/20200630-publisher-report-ndisplays.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advertiser Side Reporting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = f\"\"\"\n",
    "    SELECT\n",
    "        d.advertiser AS advertiser\n",
    "        , d.publisher AS publisher\n",
    "        , SUM(d.displays)/(1 + day(date '2020-06-17' - date '2020-06-11')) as nb_display\n",
    "    FROM\n",
    "        logs.displays d\n",
    "    WHERE\n",
    "        d.day BETWEEN '2020-06-11' AND '2020-06-17'\n",
    "        AND d.environment = 'web'\n",
    "        AND d.partner_id IN ('my_big_advertiser', 'my_medium_advertiser', 'my_small_advertiser')\n",
    "    GROUP BY\n",
    "        d.advertiser\n",
    "        , d.publisher\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = \"./data/20200630-advertiser-side-obfuscated.csv\"\n",
    "\n",
    "csv_exists = os.path.isfile(data_path)\n",
    "if csv_exists:\n",
    "    df = pd.read_csv(data_path) \n",
    "else:\n",
    "    df = querying_package.run_presto_query(query, output=data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['nb_display'] = df['nb_display'].astype('float')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Share of unreported publishers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aggregate the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp = df.groupby(['advertiser', 'nb_display'])[['publisher']].agg('count').sort_index()\n",
    "tmp['n_publishers_cum_sum'] = tmp.groupby(level=['advertiser']).cumsum()\n",
    "tmp['n_publishers_running_pct'] = tmp.groupby(level=['advertiser'])['n_publishers_cum_sum'].transform(lambda x: x / x.iloc[-1])\n",
    "tmp = tmp.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(figsize=(18, 9))\n",
    "ax.grid(False)\n",
    "sns.lineplot(data=tmp\n",
    "             , x='nb_display'\n",
    "             , y='n_publishers_running_pct'\n",
    "             , hue='advertiser', hue_order = ['Big Advertiser', 'Medium Advertiser', 'Small Advertiser']\n",
    "             , linewidth=3)\n",
    "plt.xlabel('k threshold', fontsize=16)\n",
    "plt.xscale('log')\n",
    "plt.ylabel('Share of unreported publishers', fontsize=16)\n",
    "ax.set(ylim=(-0.1, 1.1))\n",
    "plt.title('Share of unreported publishers in daily reporting with k-thresholding', fontsize=20)\n",
    "plt.gca().yaxis.set_major_formatter(ticker.PercentFormatter(xmax=1))\n",
    "\n",
    "handles, labels = ax.get_legend_handles_labels()\n",
    "ax.legend(handles=handles[1:], labels=labels[1:], prop={'size': 16})\n",
    "\n",
    "plt.axvline(1, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(1+0.05, 0.2, f\"k = 2\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(9, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(9+0.5, 0.2, f\"k = 10\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(99, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(99+5, 0.8, f\"k = 100\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(999, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(999+50, 0.8, f\"k = 1000\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(9999, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(9999+500, 0.8, f\"k = 10000\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.savefig(f'./output/20200630-advertiser-report-npublishers.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Share of unreported displays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aggregate the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['nb_display_dim'] = df['nb_display']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp = df.groupby(['advertiser', 'nb_display_dim'])[['nb_display']].agg('sum').sort_index()\n",
    "tmp['n_displays_cum_sum'] = tmp.groupby(level=['advertiser']).cumsum()\n",
    "tmp['n_displays_running_pct'] = tmp.groupby(level=['advertiser'])['n_displays_cum_sum'].transform(lambda x: x / x.iloc[-1])\n",
    "tmp = tmp.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(figsize=(18, 9))\n",
    "ax.grid(False)\n",
    "sns.lineplot(data=tmp\n",
    "             , x='nb_display_dim'\n",
    "             , y='n_displays_running_pct'\n",
    "             , hue='advertiser', hue_order = ['Big Advertiser', 'Medium Advertiser', 'Small Advertiser']\n",
    "             , linewidth=3)\n",
    "plt.xlabel('k threshold', fontsize=16)\n",
    "plt.xscale('log')\n",
    "plt.ylabel('Share of unreported displays', fontsize=16)\n",
    "ax.set(ylim=(-0.1, 1.1))\n",
    "plt.title('Share of unreported publishers in daily reporting with k-thresholding', fontsize=20)\n",
    "plt.gca().yaxis.set_major_formatter(ticker.PercentFormatter(xmax=1))\n",
    "\n",
    "handles, labels = ax.get_legend_handles_labels()\n",
    "ax.legend(handles=handles[1:], labels=labels[1:], prop={'size': 16})\n",
    "\n",
    "plt.axvline(9, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(9+0.5, 0.8, f\"k = 10\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(99, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(99+5, 0.8, f\"k = 100\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(999, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(999+50, 0.8, f\"k = 1000\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.axvline(9999, linestyle='--', color='c', linewidth=1)\n",
    "plt.text(9999+500, 0.8, f\"k = 10000\"\n",
    "         , bbox={'facecolor':'c', 'edgecolor': 'c', 'pad':4, 'alpha': 0.7}, zorder=12, fontsize=16)\n",
    "\n",
    "plt.savefig(f'./output/20200630-advertiser-report-ndisplays.png', bbox_inches='tight')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}