grupawp/paapi

Protected Audience API tests

PAA (Protected Audience API) test on Wirtualna Polska Media Inventory

Table of Contents:

• Description
• Publisher ad stack overview
• PAA test settings
• PAA test diagram
• PAA test results

Description

The following document describes test setup and results on tech.wp.pl website, where the publisher enables PAA auctions on select adslots.

Intent to participate in PAA auction (in the form of componentAuction config) is gathered using header bidding framework. After header bidding auction ends, the publisher will execute PAA auction (using navigator.runAdAuction method) and score bids generated by buyers (score is proportional to bid value, unless bid value is below floorprice - in which case score is set to zero). Auction results (Uniform Resource Names) are stored (along with auction floorprice) and used when ad selection logic is executed. 

All adslots are executed based on user navigation and viewability thresholds; adslot execution is delayed by both auctions (Header Bidding and PAA). 

Publisher ad stack - a short overview

• Publisher uses it's own adserver that connects via OpenRTB protocol to internal and external (server-to-server) bidders
• Adserver is called once per page view (SRA architecture)
• All adsever responses contain ad code, buyer info and cpm value
• In addition, publisher uses header bidding framework (Prebid.js)
• The publisher uses Lazy Bidding for header bidding auction and Lazy Loading for adslot execution
• Ad selection is done by comparing header bidding cpm with ad server cpm
• Finally adslot can send request to Google Ad Manager using winning cpm as floor price

Protected Audience API test settings

• Publisher checks availability of PAA (by checking support for runAdAuction method)
• For PAA-enabled inventory publisher selects five adslots (three on desktop, two on mobile) that can execute PAA auction
• Header bidding buyers can indicate intent to participate in PAA auction by sending componentAuctions
• At the moment only one buyer - RTB House - participates in the test
• ComponentAuctions are gathered by Prebid.js and send to publisher (this is a modification of fledgeForGpt module)
• ComponentAuctions are gathered for 50% of PAA enabled inventory, but auction is not conducted (this is a control group)
• For the remaining 50% of the inventory publisher waits for header bidding auction to end and then executes PAA auction
• Results of PAA auction (URNs) are stored and can be used when adslot has started
• Adslots - as described in "publisher ad stack" - start asynchronously based on viewability thresholds
• Adslot execution is paused until PAA promise is resolved (or 1000ms timeout is triggered)
• After PAA promise is resolved call to Google Ad Manager can be made using available highest ad value as floor price
• Additionally Google receives PAA auction status in targeting (whether auction was started, returned no result or is still ongoing). This allows the publisher to ascertain how PAA affects Ad Manager metrics

Publisher collects the following data for the test (in addition to standard ad metrics):

• Per page view:
  • PAA availability
  • Test group
• Per PAA auction
  • Auction events (started, result available, timed out)
  • Auction duration
  • Auction floorprice
  • Auction wins (seller.js -> reportResult → sendReportTo method) 

Additionally publisher collects adslot metrics (ads available / ads delivered / ads viewed) for three groups:

• Impressions with PAA support, but no PAA auctions (control group)
• Impressions with PAA support and PAA auctions that have no winning ads. Note that the publisher is unable to discern between auctions with no interest groups and auctions that have interest groups with ads below the floor price
• Impressions with PAA support and PAA auctions that have winning ads

Protected Audience API test diagram

Protected Audience API test results

PAA availability

At the beginning of the test in July PAA was available only for <10% of impressions. Since September the support has increased massively and currently it stands at 55% overall.

Inventory Type	PAA support
Desktop	42%
Mobile	68%
Overall	55%

PAA bid rate and timeout rate

Buyer will indicate willingness to participate in auction for almost all of PAA-enabled inventory.

Win rate in PAA is very low (sub 1%), but we cannot determine whether this is due to floorprices or lack of Interest Groups for the users.

Average auction lifetime is low, but this is only due to low win rate (see below).

Site Version	AdSlot	Header Bidding with PAA	PAA with result	PAA with timeout	Avg auction duration [ms]
desktop	11	85%	0,13%	4,96%	48
desktop	36	84%	0,08%	4,79%	128
desktop	93	87%	0,10%	16,39%	66
mobile	1	94%	0,14%	6,52%	172
mobile	3	94%	0,14%	0,05%	178
mobile	5	95%	0,11%	7,11%	50

PAA auction duration

Again, it is not possible to discern between auctions that are empty (no interest groups to bid) and auctions that have interest groups, but results are below floorprice.

PAA result	Average response time [ms]	Median response time [ms]	80th percentile of response time [ms]
No ad	155	52	161
Ad	1356	733	1773

Notes on test results

After extensive testing we can point out three main issues:

• Debugging PAA auctions
• Available signals for PAA auctions / creatives
• Extended auction execution time / sequential nature of PAA

Debugging PAA auctions

In our experience, debugging PAA auctions is too complicated. When it comes to available information:

• joining an Interest Group is displayed in Application → Storage → Interest Group tab
• starting auction will show which groups are available (loaded) in the above tab
• auction events (generate bid, score ad, report result) are not registered in Interest Group tab. Calls to worklet can be recorded in in Performance tab
• ads picked and bids scored are not available anywhere (that's why we need option to switch to debug-seller.js script)
• reporting events (sendReportTo) are not available either in Interest Group or in Performance tabs

The above significantly impairs ability to debug PAA auction that is currently running in the browser. 

Available signals for PAA auctions / creatives

As mentioned in test results, currently there is no way to discern between:

• an "empty" PAA auction (no interest groups)
• PAA auction where interest groups did not not generate any bids
• PAA auction with bids with zero score (bids that are below the floor price and were rejected by the seller)

Furthermore, bid won and bid delivery events are combined into one reportResult event (for seller reportResult is called at the end of the auction, but the endpoint registered in the reportEvent method will be called when the opaque iframe / fenced frame containing the  winning bid is loaded). 

This means that - in the ad stack where auctions and adslots are executed asynchronously (as in this test) - the Publisher  is not able to tell the difference between:

• auctions with no results
• auctions that have not resolved before adslot was loaded
• auctions that resolved on time, but results were not used for a different reason

In contrast -  in the standard header bidding framework -  the Publisher is notified on the following events:

• auction start
• bid request / response (per bidder)
• auction end
• bid won
• bid delivery started (creative appended to the adslot)
• bid delivery ended (creative fully loaded)
• bid viewable

The last four events are especially crucial for optimizing and detecting errors in the header bidding environment.

Note that the issue is not with the data availability for each event (and whether the data breaks k-anonymity), but with the availability of the event itself. 

Extended auction execution time / sequential nature of PAA

At the moment PAA auction execution time - on auctions that have returned an ad - is reaching unacceptable values. Values - which due to sequential nature of PAA (PAA does not return ad value and has to be executed afer all other demand sources) - cause noticeable delays to ad rendering. That has a strong negative impact on ad performance and viewability. It also hurts user experience.

PAA should be executed in parallell to other demand sources and provide information necessary for the publisher to select best / winning ad. In such case the ad stack would look as follows:

paapi's Issues

IG Count Question

Am I correct in understanding that there is not a known IG count per auction?