Tutorial

Whether you're a seasoned bug hunter or just starting out, this tutorial will make sure you get a hold of the BugHog basics. At the end, you will have utilized BugHog to identify the lifecycle of a publicly disclosed bug, by leveraging its proof of concept in more than 100 automated experiments.

Before we set out on our 30-minute adventure, make sure you have access to the following prerequisites:

• Docker installation
• Internet connection

Adventure time! 🚀

As you explore the world of web browser, you stumble upon this intriguing artifact - a dead bug in Chromium. I say dead because this particular bug has been fixed according to the linked bug report. Moreover, it's juicy, since the reporter was awarded a bug bounty of $1000! Such bounties are not uncommon when bugs impact security policies, as is the case here, given that this bug acts as a bypass for the Content Security Policy (CSP). This piques the curiosity of your inner entomologist, making your mind grind on some very essential questions.

Where did the bug come from? How long has it been around? Is it truly deceased?

Alas, the aforementioned bug report offers no solace, leaving us short of the answers to these fundamental inquiries. But fear not, for BugHog will emerge as your indispensable ally in this quest for truth! So move aside Pumba, we now have our own bug-obsessed hog to relieve us of our worries! Let the adventure commence!

Obtaining BugHog 🐗

First, let's summon the BugHog. The fastest way is cloning this repository and downloading the pre-built Docker images. These commands will set you up:

git clone https://github.com/DistriNet/BugHog.git BugHog  # Clone repository
cd BugHog
./scripts/pull.sh   # Download images
./scripts/start.sh  # Run BugHog

Note

If you're using WSL on Windows, it's important to clone the repository on the WSL file system.

If all goes well, after a few minutes, you should be able to access BugHog's user interface by surfing to http://localhost:80 with your preferred browser. While BugHog will work out of the box, some aspects can be configured in more detail if desired. For now, we will use its default setup.

Note

If you're hosting BugHog on a remote server, use its IP or domain instead of localhost.

If something goes wrong, check out the Troubleshooting section for help.

Using BugHog for a general analysis 🔬

Let's start simple: first, we just want to get a general idea of our dead bug's lifecycle!

Dissecting the bug

Before BugHog can track a bug, it must be aware of its intricacies, or in other words, it needs to know how the bug is reproduced. BugHog leverages dynamic analysis to infer whether a bug can be reproduced at a certain moment in the development history of a browser, and as such, we will need to be able to reproduce the bug in browser binaries. Bug reports describe how to reproduce the reported bug, often in specific steps, which is called a proof of concept (PoC).

Indeed, our bug report does exactly this, but still, every bug that is to be evaluated needs its PoC to be integrated into BugHog. To save you valuable time, this has already been done beforehand; this particular bug has been integrated by the files in this folder. As you can see, it contains the web page that reproduces the bug, based on the report's PoC.

Note

BugHog allows adding new experiments through its web UI since v2.2.0. However, more information about the underlying file structure can be found here.

Bisecting the bug

Now that we're sure BugHog is familiar with the bug, it's almost time to unleash it! We just have to give it a bit more guidance and off we go!

The user interface presented at http://localhost:80 might look a bit overwhelming at first, but for this adventure, we'll only need part of its functionalities.

Tip

If at any point you are not sure what something means, you can hoover over the info signs in the web UI.

1. Select the browser

   First, we will indicate which browser we want to evaluate. Currently, BugHog supports Chromium and Firefox. Since it's a Chromium bug, let's choose Chromium for now; check chromium in the Browser pane.

2. Select the history segment

   How far do we want to go back in time? And do we want to only evaluate release binaries or even the revision binaries in between? Let's keep it straightforward and run our evaluation for all supported release binaries, which is the default setting (you don't have to change anything here).

   This range off Chromium binaries encompasses over a decade of browser development and reaches back to binaries released in the year 2012. So, in a way, you could consider BugHog as a time-traveling hog tracing bugs throughout history. That sounds like a query for a generative AI image!

3. Select the bug

   Next, select the bug that we want to trace by selecting the CSP project in the Experiments pane. Locate our bug in the list, identified as c971231 (here, the 'c' indicates that this is a Chromium bug, followed by the bug's id), and tick its box to indicate that this experiment should be included in the evaluation. Now BugHog knows which bug we want to track.

Tip

You can edit every experiment through the web UI by clicking the edit symbol of an experiment. This is not required in this tutorial, though.

4. Select which experiment to follow

   The Results pane will display the progress and intermediary results of the evaluation. Use the dropdown menu to select c971231. This pane will now be updated with the progress and results of the c971231 bug evaluation.

5. Let loose the BugHog!

   Simply press the big ol' green start button to begin trackin'! Once clicked, you should notice that this green button makes way for a yellow and red button, and that the Status indicates that BugHog is running.

Note

We did not consider the Advanced options pane at the bottom, where more detailed settings can be modified. You can find more information about all options by hoovering over the information signs.

Analyzing our results

You'll be able to track the pinpointing progress throughout the evaluation by monitoring the Results pane. Here, you'll see the number of experiments displayed, as well as a Gantt chart. Let's return in 5 minutes, this should give BugHog time to conduct at least 35 experiments when running with 8 parallel containers. Allow for a bit more time when running with less.

A few moments later...

Welcome back! If all went well, you should see that more or less 35 green squares have turned up the Gantt chart of the Results pane. Click the yellow button Stop gracefully to give BugHog a pat on the back. This will make it finish the ongoing experiments and stop further tracking (this can take a minute).

Every green square represents a release binary that has been evaluated against the selected experiment. The table below represents all binary segments identified by BugHog. While not all binaries may have been evaluated in your BugHog session yet, check whether your results fit into the segments displayed in this table.

Binary segment	Context
v20 - v25	🚫 These binaries do not support CSP yet, so our experiment's CSP header is ignored. This was demonstrated for explanatory purposes in this tutorial, but you can avoid evaluating out-of-scope binaries by adjusting the Browser version range slider to the required range.
v26 - v39	✅ The binaries in this segment correctly handle our experiment; the PoC is not reproduced.
v40 - v62	🐛 Our PoC is reproduced in all these releases. This should not have happened, and as such, this is a bug!
v63 - v73	✅ The bug has been fixed in this release range.
v74 - v91	🐛 Damn it! The bug is again affecting release binaries!
v92 - ...	✅ And the bug has been fixed again!

As you can see, our bug was introduced, fixed, reintroduced and eventually fixed again. The reintroduction even affected multiple release versions, accounting for about two years of exposure! So, what else can we learn from this?

Using BugHog for targeted tracking 🎯

Alright, BugHog has helped us understand our bug's lifecycle better, but there's more to uncover! What about its true origins? What changes to the code base introduced this bug the first and second time? We can use BugHog to zoom in on these parts of the lifecycle.

Deep search

In the previous part of this adventure, we didn't unleash the full power of BugHog; we only looked at binary releases. But that's about to change with just a single click! Go ahead and tick the Deep search box in the Browser version range pane.

This option will make BugHog consider revision binaries too, enabling us to search for bug introductions at the level of individual commits in the browser's version control system (like Git for Chrome or Mercurial for Firefox).

Now, let's shorten our evaluation range to save some CPU cycles. Our preliminary results indicated that the bug was first introduced between v39 and v40, so we'll set the lower bound to v39. Similarly, we'll set the upper bound to v91, after which the bug seems to have been fixed.

Tip

While it seems straightforward to ignore the outer version ranges here, it might still be useful to perform a deep search over the full version range. Bugs can sometimes be introduced and fixed between two release versions. If this happened between v35 and v36, for instance, we would miss it now.

Perfect! Now let's click the green start button again and give BugHog 15 minutes to do its thing.

Note

By default, BugHog first evaluates 50 binaries within the selected version range in a uniform binary sequence. After this initial sequence, BugHog starts focusing on shifts in reproducibility, using binary search to identify the code changes or ranges where the bug was introduced. Increasing the Sequence limit in the Advanced options pane can make your evaluation more thorough.

Analyzing the results

Feel free to hit the Stop gracefully button again if BugHog hasn't finished after 15 minutes.

Now, you should see new blue dots next to the green squares in the Gantt chart. You've guessed it: while the green squares represent (approximations of) release versions, the blue dots represent code revisions.

Tip

If the Gantt chart gets a bit crowded, try navigating it by dragging and scrolling to clear things up.

Normally, BugHog should have already identified the code revision that introduced our bug for the first time. By zooming in between v39 and v40, you'll see a bunch of blue dots. The first dot where the bug shifts to Reproducible is the actual revision that caused the bug. By hovering over this dot, the Gantt chart should display the following information:

Revision Number	Browser Version
294410	39

Since the Chromium source code is publicly available, we can dive deeper into this revision. Clicking the blue dot will open the public revision information page in a new tab. This revision represents a "Blink roll", where an update to the underlying browser engine Blink is integrated into the Chromium code base. So, it seems the bug stems from a change in the Blink source code.

Let's investigate further by clicking on the first link in the commit message, which will take us to the exact set of code changes included in this Blink update. Reviewing the commit message titles, we find two commits related to CSP code, but the one lower down is the one we're interested in. This commit modifies core CSP functionality to address a vulnerability mentioned in bug 411600, which is referenced in the revision message.

As such, we’ve discovered that our bug was introduced by developers working to fix another vulnerability!

Concluding the adventure 🎉

Congratulations!

You have successfully bisected the introduction of a bug!

What did we learn by pinpointing thus far?

• The bug is indeed dead, as BugHog could not reproduce it in recent browser binaries.
• It was introduced with revision 294410, which tried to fix another vulnerability.
• Although the bug was fixed at a later time, it was eventually reintroduced, impacting the browser again for quite some time (two years!).

If you're interested in discovering how the bug was reintroduced, feel free to continue pinpointing! Or maybe you want to find out whether this bug also affect(ed) Firefox? There seems to be no public report describing this bug on Firefox's bug reporting platform, but who knows? 👀

Note, however, that exact pinpointing of bug lifecycles in Firefox may pose more challenges, because of the limited availability of revision binaries. This is solved by building the missing binaries yourself and providing them to BugHog, which will be covered in another tutorial.

While this tutorial ends here, you can always leave a message in case of issues, difficulties or questions!

Lastly, whenever you want to stop BugHog entirely, use the following command inside its project folder:

./scripts/stop.sh

Happy hogging! 🐛 🐗