decaffeinated // posts

David Heinemeier Hansson has asked the question: who benefits by overstating complexity? I was reminded of DHH as it's become known that Microsoft (the steward of AppNexus/Xandr) plans to deprecate their hosted, public Prebid Cache.

Prebid Cache is a core dependency[1] of video ads (and to a now-lesser extent, AMP) served via header bidding to GAM. I never had to think about Prebid Cache more than a configuration option to Prebid.js, but I was left wondering: do publishers really need to run into the arms of another company to replace Microsoft?

Reviewing the Code

Prebid Cache is a pretty simple design. Serve requests from the internet with Go and provide a few different NoSQL backend options to store cached data. (You can tell that it's an adtech product as Aerospike is supported in addition to memcached and redis.)

There is a Dockerfile that makes it easy to deploy and run Prebid Cache without worrying too much about the internals. You can set your configuration options via environment variables. With some small tweaks to that Dockerfile and supporting scripts, we're only a few steps from production.

Deploying Prebid Cache

Set Up Dokku

In our mission of avoiding dependencies on Big Tech I'm going to deploy this to my own server, not AWS ECS. Dokku is as close as I've found to the simplicity of a Heroku-like deployment experience, with none of the associated cost.

On the server:

# download the installation script
wget -NP . https://dokku.com/bootstrap.sh
# run the installer
sudo DOKKU_TAG=v0.37.2 bash bootstrap.sh
# Configure your server domain, not mine :) make sure you update your DNS to point this this server if not already
dokku domains:set-global decaffeinated.io
# and your ssh key to the dokku user
PUBLIC_KEY="your-public-key-contents-here"
echo "$PUBLIC_KEY" | dokku ssh-keys:add admin
dokku apps:create prebid-cache

Once Dokku is installed, we can work to get Prebid Cache live.

Create Prebid Cache App

I'm going to use Redis as my backend because:

a) it's bulletproof

b) I've used it in production for years

c) The plugin is standard in Dokku and I can run it on the same machine

# optional: use subdomain for cache
dokku domains:add prebid-cache prebid-cache.decaffeinated.io
# create redis plugin
dokku redis:create prebid-cache-redis
# link app to plugin
dokku redis:link prebid-cache-redis prebid-cache

# map application ports to web serving
dokku ports:set prebid-cache https:443:2424
dokku proxy:ports-set prebid-cache http:80:2424

# optional -- if using Cloudflare it may be easier to set up SSL there
dokku letsencrypt:set prebid-cache email [email protected]
dokku letsencrypt:enable prebid-cache
dokku letsencrypt:cron-job --add

Now is a good time to make those small changes to our Dockerfile. I deployed to a cheap ARM server, so I had to modify the standard file to support ARM instead of x86 -- I have a PR open on the Github to solve this automatically.

Let's clone the repo and make changes to the Dockerfile:

git clone https://github.com/prebid/prebid-cache.git && cd prebid-cache

# only if using an ARM server
-ENV GO_INSTALLER=go1.19.5.linux-amd64.tar.gz
+ENV GO_INSTALLER=go1.19.5.linux-arm64.tar.gz

And to support running in Dokku:

-ENTRYPOINT ["/usr/local/bin/prebid-cache"]
+ENTRYPOINT ["/docker-entrypoint.sh"]

The contents of that docker-entrypoint.sh file are in the appendix below[2]. The entrypoint will automatically parse the redis URL that the Dokku plugin provides into Prebid Cache-friendly arguments.

Finally, commit your changes and push up Prebid Cache to Dokku

git remote add dokku [email protected]:prebid-cache
git push dokku main

If all goes well, you'll now have Prebid Cache serving traffic and running on the domain/subdomain you specified.

Hardening

Can our little install actually handle production workloads?

Without any scaling, our instance gracefully handles ~110 RPS of 1:2 write/read workload:

  █ TOTAL RESULTS 

    checks_total.......: 6497    108.248731/s
    checks_succeeded...: 100.00% 6497 out of 6497
    checks_failed......: 0.00%   0 out of 6497

    ✓ POST 200
    ✓ GET 200

    HTTP
    http_req_duration..............: avg=27.25ms min=12.1ms  med=24.57ms max=419.25ms p(90)=36.6ms  p(95)=44.26ms
      { expected_response:true }...: avg=27.25ms min=12.1ms  med=24.57ms max=419.25ms p(90)=36.6ms  p(95)=44.26ms
    http_req_failed................: 0.00%  0 out of 6497
    http_reqs......................: 6497   108.248731/s

My server is decent. 4 ARM cores, < 25 GB RAM. It's the equivalent of an okay laptop. And it's serving other applications, like this website. The dedicated cost isn't more than $100 a year, and probably about half that. For comparison, I've seen providers offer discounted lower throughput than what we just achieved for $2400!

To support a real-world, scaled scenario, I would want to get some monitoring in place and scale up our processing to load-balance (as easy as dokku ps:scale prebid-cache web=2), but this should be a low-drama, low-cost component of our infrastructure.

Prebid Cache doesn't require enterprise complexity or enterprise pricing. With about an hour of setup, a basic server, and standard open-source tools, you can self-host a production-ready instance for a fraction of vendor costs. Sometimes the best response to deprecated services is recognizing you never needed an intermediary in the first place.

Appendix

1. There is some interesting work ongoing to try and move the caching entirely to the client-side, but it's unclear if that will be successful.

#!/bin/sh
set -e

# Parse REDIS_URL if it exists
# Format: redis://[:password@]host:port[/db]
if [ -n "$REDIS_URL" ]; then
  echo "Parsing REDIS_URL..."
  
  # Remove redis:// prefix
  REDIS_CLEAN=$(echo $REDIS_URL | sed 's|redis://||')
  
  # Check if password exists (contains @)
  if echo $REDIS_CLEAN | grep -q '@'; then
    # Extract password (everything before @, after the first :)
    export PBC_BACKEND_REDIS_PASSWORD=$(echo $REDIS_CLEAN | sed -E 's|^:([^@]+)@.*|\1|')
    # Get host:port/db part (everything after @)
    HOST_PART=$(echo $REDIS_CLEAN | sed 's|^[^@]*@||')
  else
    # No password, entire string is host:port/db
    HOST_PART=$REDIS_CLEAN
  fi
  
  # Extract host (before :)
  export PBC_BACKEND_REDIS_HOST=$(echo $HOST_PART | cut -d: -f1)
  
  # Extract port and optional db
  PORT_DB=$(echo $HOST_PART | cut -d: -f2)
  
  if echo $PORT_DB | grep -q '/'; then
    # Has database number
    export PBC_BACKEND_REDIS_PORT=$(echo $PORT_DB | cut -d/ -f1)
    export PBC_BACKEND_REDIS_DB=$(echo $PORT_DB | cut -d/ -f2)
  else
    # No database number
    export PBC_BACKEND_REDIS_PORT=$PORT_DB
    export PBC_BACKEND_REDIS_DB=0
  fi
  
  echo "Redis configured: host=$PBC_BACKEND_REDIS_HOST port=$PBC_BACKEND_REDIS_PORT db=$PBC_BACKEND_REDIS_DB"
fi

# Execute the main application
exec ./prebid-cache "$@"