Skip to content
This repository has been archived by the owner on Jun 14, 2024. It is now read-only.

Home

Jump to bottom
Osvaldo Pinali Doederlein edited this page Jul 17, 2018 · 18 revisions

OpenRTB Core

This quick tutorial guides you into programming with the OpenRTB Core library.

The OpenRTB specification

This library implements the protocol / data model from the latest OpenRTB Specification, so reading that should be your first step; it's also a good introduction to the concepts of real-time bidding.

OpenRTB is only specified in terms of JSON messages, so an implementation has to provide bindings for specific programming languages so you can create and manipulate its message objects in a convenient way, i.e. not needing to deal with the raw JSON representation. We do that by first translating the specification to a Protocol Buffer descriptor, which can be used to generate model classes. This library will only build a Java model out-of-the-box, but you can easily use the same Protobuf descriptor to generate the model code for many other languages. (Other features though, are only available for Java or other JVM-based languages.)

The RTB Model

Protobuf is often used to create very efficient, evolvable binary protocols or data storage formats, but in this case we're only interested in its ability to produce a very rich model. Here's a snippet that uses the Java model:

BidRequest request = BidRequest.newBuilder()
    .setId("1")
    .addImp(Imp.newBuilder()
        .setId("1")
        .setBidfloor(4000)
        .setVideo(Video.newBuilder()
            .setLinearity(VideoLinearity.LINEAR)
            .addProtocols(Protocol.VAST_3_0)
            .setW(640)
            .setH(480)))
    .build();

Some qualities of this generated model:

  • Fully static-typed (including extensions, commented later)
  • All message types (e.g. BidRequest) are immutable, paired by fluent Builders
  • All message types, getters and setters carry documentation extracted from the OpenRTB specification
  • Present/absent state for all properties, including those of primitive type, with hasXxx() methods
  • Other conveniences, check the Protobuf API Reference

Extensions

OpenRTB is a best-effort at a common RTB protocol, but no SSP or DSP platform uses only "pure OpenRTB". RTB protocols are varied and fast-changing, so everyone has at least a few fields or objects that are not yet available in the OpenRTB spec. That's why OpenRTB allows platforms to add extensions, in the form of arbitrary objects/fields inside "ext" properties of most objects, like this example:

"geo": { "country": "USA", "city": "Newer York", "zip": "102879",
         "ext": { "sa-planet": "Mars" }
}

The snippet above is part of a BidRequest message, in its JSON representation. Unfortunately for the SpaceAds SSP, OpenRTB's Geo object doesn't support a planet field, necessary for geotargeting users or devices from our Martian colonies. This hypothetical SSP from the future would need an extension for this information, at least until a new OpenRTB release supports it. (Field name prefixes are common practice, and highly recommended, so messages might combine extension fields from multiple parties with less risk of conflict; that's why the example uses sa-planet, "sa-" standing for SpaceAds.)

The library supports OpenRTB extension conveniently and safely, via Protobuf Extensions:

BidRequest request = BidRequest.newBuilder()
    // ... add Imp, etc.
    .addDevice(Device.newBuilder()
        .setModel("Nexus 85")
        // ... other standard Device fields
        .setGeo(Geo.newBuilder()
            .setCountry("USA")
            .setCity("Newer York")
            .zetZip("102879")
            .setExtension(SpaceAdsExt.planet, "Mars")))
    .build();

Here's how extensions work: the SpaceAds SSP provides a separate extension library, also Protobuf-generated, which only defines the extensions it contributes to specific OpenRTB extension points. In the generated model class like SpaceAdsExt, each of these extensions will have a key like 'SpaceAdsExt.planet'. In the core OpenRTB object like BidRequest.Geo, which doesn't know anything about extensions, you have to use a generic method setExtension() to add extension data. Notice however that this setter is static-typed: it will only accept matching (key, value) pairs, and both the key and value have to be declared as compatible with the BidRequest.Geo.ext extension point. This mechanism combines modularity (allowing extensions to be provided by third-party libraries without any change to the core OpenRTB library) with type-safety (any incorrect use of extension will not compile).

Note: Reading binary protobufs require some extra work, so the desserializer will handle extensions correctly:

    ExtensionRegistry reg = ExtensionRegistry.newInstance();
    SpaceAdsExt.registerAllExtensions(reg);
    ...
    BidRequest req = BidRequest.parseFrom(bytes, reg);

JSON Support

OpenRTB messages will typically be received and sent in their JSON format. Protobuf's generated code doesn't support JSON serialization; there are third-party libraries that do that, but they don't help us because support for extensions creates serious problems for any automatic JSON serializer (the content of the ext fields is not typed in the container object, and it may be a combination of fields contributed by several independent extensions). The library solves this problem by providing a custom JSON serializer, based on the popular Jackson library. (Your programs only need the Jackson dependency if they use the JSON serializer component.) Sample usage:

OpenRtbJsonFactory openrtbJson = OpenRtbJsonFactory.create()
    .register(new SpaceAdsGeoReader(), BidRequest.Geo.Builder.class)
    .register(new SpaceAdsGeoWriter(), String.class, BidRequest.Geo.class)
    .create();
// How to serialize
ByteString jsonReq = openrtbJson.newWriter().writeBidRequest(request);
// How to desserialize
BidResponse resp = openrtbJson.newReader().readBidResponse(jsonResp);

You start by creating an OpenRtbJsonFactory, which can be optionally configured with any number of "extension readers" and "extension writers". In the example, the SpaceAds SSP provides a library that contains classes like SpaceAdsGeoReader and SpaceAdsGeoWriter. The example above has a single extension field of scalar type (a String), but extensions can also define custom messages, repeated field... check out the included unit tests for more examples. A real-world scenario may need a large number of these register() calls, but the third-party library might provide a utility method that makes all necessary register calls to an OpenRtbJsonFactory.

Macro Processing

A common usage scenario for the RTB model is that of a bidder or DSP that receives a BidRequest and replies a BidResponse. The response is simple in number of objects/fields, but it may contain fields which content is complex (notably Bid.adm = ad markup) or derived from information from the corresponding request or even from other response fields. This problem is well-fit for macros, so this library offers a SnippetProcessor API that performs post-processing of a BidResponse.Builder (notice the Builder here, since we need the response's mutable representation for postprocessing). An example will make this clear:

BidResponse.Builder responseBuilder = BidResponse.newBuilder()
    .addSeatbid(SeatBid.newBuilder()
        .addBid(Bid.newBuilder()
            .setAdid("ad-1234567")
            .setImpid(imp.getId())
            .setPrice(1.2)
            .setAdm("<a href='http://test.com&adid=%{${AUCTION_AD_ID}}%'>"
                  + "<img src='http://cdn.com/my-ad.jpg'/>"
                  + "<img src='http://mybidder.com/impression-pixel'>"
                  + "</a>")))));
SnippetProcessor proc = new OpenRtbSnippetProcessor();
proc.process(request, responseBuilder);
BidResponse response = responseBuilder.build();

Above we're creating a BidResponse corresponding to a specific BidRequest, containing a Bid corresponding to a specific Imp(ression) from that request. The HTML markup for this bid contains a parameter adid in the click-through URL, which value will be the same as the Bid.adid. You could just repeat the value ad-1234567 inside the markup, but these values are typically not constants; they are often calculated, or come from some campaign data. Now you don't want to create a complex HTML string with a spaghetti of concatenations, do you? (The example above is simple, but real ads can have much bigger HTML snippets with several dynamic parameters.) The solution is using the ${AUCTION_AD_ID} macro, which will be expanded to the content of Bid.adid by the process() call.

Notice that you actually use an OpenRtbSnippetProcessor, which supports macros defined by the OpenRTB specification. Third-party libraries can provide further subclasses to support extended macros. The base SnippetProcessor includes a single non-OpenRTB feature, because it's too useful: the syntax %{...}% will URL-escape its contents, which is often necessary in ad markup. The example above puts the markup's ${AUCTION_AD_ID} inside %{...}% because escaping may be necessary if our adid values are arbitrary strings, possibly containing non-URL-safe characters.

When should you execute this macro expansion? it may not be necessary at all, if you're sending a BidResponse to a native-OpenRTB SSP/DSP that's expected to handle these macros. (In fact you could use this library to implement an exchange or other back-end system that receives responses and expands their macros.) So, you only need to preprocess macros in some circumstances:

  • If the BidResponse will be translated to another protocol, to be sent to a non-OpenRTB receiver.
  • If you want to benefit from extended macros, including the %{...}% syntax.

Native Ads

Native Ads are specified by the standards OpenRTB 2.3 & OpenRTB Native 1.0. For the most part, native ads are just additional objects and fields in the RTB model; but there are some special considerations.

The Native model is included in the same descriptor (openrtb.proto), which allows reuse of shared elements. This is also necessary because, despite OpenRTB specs being loosely-coupled ("native markup" is added to the core JSON messages via embedded string fields), our library supports an optional static-typed API that's safer, more readable, efficient and convenient. Here's how you create a request:

OpenRtb.BidRequest.newBuilder()
    .addImp(OpenRtb.BidRequest.Imp.newBuilder()
        .setId("imp3")
        .setNative(OpenRtb.BidRequest.Imp.Native.newBuilder()
            .setRequestNative(NativeRequest.newBuilder()
                .setLayout(LayoutID.APP_WALL)
                .setAdunit(AdunitID.PROMOTED_LISTING)
                .setPlcmtcnt(4)
                .setSeq(5)
                .addAssets(NativeRequest.Asset.newBuilder()
                    .setId(1)
                    .setReq(true)
                    .setTitle(NativeRequest.Asset.Title.newBuilder()
                        .setLen(100))))))
    .build();

This is not an extension / incompatibility with the OpenRTB specification; when you serialize this message to JSON, that will produce the expected output where request's imp.native.request field is a string like this:

{"id":"imp3","native":{"request":"{\"ver\":\"1.0\",...},...},...}"

Notice the request field needs lots of escaping, because we're embedding one JSON document inside another JSON document. But the library's JSON serializer / desserializer hides all this nastiness. Another advantage of the static-typed model is that we may not need any JSON serialization or escaping, if the messages will be being translated to another protocol for some non-OpenRTB exchange or if an exchange supports OpenRTB/Protobuf protocol.

DoubleClick Ad Exchange supports OpenRTB/Protobuf as an alternative to the standard OpenRTB/JSON encoding, using the openrtb.proto from this library. We're keeping this proto descriptor (in fact this entire library) free of any DoubleClick-specific extensions, inviting any other platforms/products that want "binary OpenRTB" to adopt this Protobuf-based solution.

The response model is similar: in addition to the standard Bid.adm field, which can now contain escaped-JSON "native markup", you can use the adm_native field, which will be transparently converted by the serializer so the JSON representation is always compliant with the standard. This may once again avoid unnecessary JSON (de)serialization/escaping if you never really need the JSON representation.

Other Utilities

This library also offers some small conveniences in the util package:

  • OpenRtbValidator: inspects a pair of request/response and finds many potential errors in the response
  • OpenRtbUtils: methods to lookup, filter and update elements of requests and responses

Next Steps

The openrtb library is a central building block for RTB systems, but building any of that (a bidder, DSP, or other OpenRTB-related application) is typically a bigger puzzle that benefits from additional pieces.

  • The openrtb-doubleclick library is a companion open source project that provides extensive support for the DoubleClick Ad Exchange platform. This includes DoubleClick-specific conveniences like validation and crypto, and some OpenRTB integration.
  • Open Bidder is a complete toolkit for building scalable, full-featured RTB bidders, optimized for the Google Cloud Platform and DoubleClick but also extensible to other platforms and exchanges.
Clone this wiki locally