Overview

This post will cover some details behind the recent Grafana vulnerability (CVE-2021-43798), which is a directory traversal bug allowing unauthenticated attackers to read files on the target server filesystem. This post will also discuss some real world scenario and attack surface of the Grafana.

Brief Analysis on the Root Cause

The detailed analysis can be found at the author’s blog here, I will only briefly cover it.

All API routes were defined in pkg/api/api.go , some require authentication like below:

	r.Get("/plugins", reqSignedIn, hs.Index)
	r.Get("/plugins/:id/", reqSignedIn, hs.Index)
	r.Get("/plugins/:id/edit", reqSignedIn, hs.Index) // deprecated
	r.Get("/plugins/:id/page/:page", reqSignedIn, hs.Index)

While some does not require signed in, like below:

	// expose plugin file system assets
	r.Get("/public/plugins/:pluginId/*", hs.getPluginAssets)

For the route at /public/plugins/:pluginId/*, it is handled by hs.getPluginAssets, which is defined in pkg/api/plugins.go:

// getPluginAssets returns public plugin assets (images, JS, etc.)
//
// /public/plugins/:pluginId/*
func (hs *HTTPServer) getPluginAssets(c *models.ReqContext) {
	pluginID := web.Params(c.Req)[":pluginId"]
	plugin, exists := hs.pluginStore.Plugin(c.Req.Context(), pluginID)
	if !exists {
		c.JsonApiErr(404, "Plugin not found", nil)
		return
	}

	requestedFile := filepath.Clean(web.Params(c.Req)["*"])
	pluginFilePath := filepath.Join(plugin.PluginDir, requestedFile)

	if !plugin.IncludedInSignature(requestedFile) {
		hs.log.Warn("Access to requested plugin file will be forbidden in upcoming Grafana versions as the file "+
			"is not included in the plugin signature", "file", requestedFile)
	}

	// It's safe to ignore gosec warning G304 since we already clean the requested file path and subsequently
	// use this with a prefix of the plugin's directory, which is set during plugin loading
	// nolint:gosec
	f, err := os.Open(pluginFilePath)
    
    the rest are Omitted

Line 5 is retrieving /public/plugins/(.*) as the pluginId, then pass to line 12 filepath.Clean to do sanitization, and concatenate in line 13, finally passed to os.Open in line 23 to read the contents.

The most interesting part is the comment at line 20:

// It’s safe to ignore gosec warning G304 since we already clean the requested file path and subsequently

If we check the document on the usage of filepath.Clean:

Point 4 is worthy to take note.

replace “/..” by “/” at the beginning of a path

What if the path does not start with /..? we can try it out:

It seems that filepath.Clean is not working as what the developers expect it to do, which leads to directory traversal and subsequently arbitrary file read.

It can be replicated in the docker environment as shown below (take note that the plugin should exist, otherwise you will get Plugin not found error. Luckily, Grafana has come with some default plugins. In the screenshot below, I am using Grafana’s welcome plugin):

Nginx Reverse Proxy Bypass

On the day this vulnerability is getting hot amongst the security researchers (around 7th Dec, 2021), the vulnerability author posted a tweet as below:

But one day later, he retweeted:

So does Nginx help? we can set up the environment and try:

We are getting a 400 bad request.

The reason is simple: Nginx will do path normalization before it forwards the request to the backend. If the normalized URI is requesting beyond the web root directory, it will simply returns 400 bad request.

In the request above, /public/plugins/welcome/../../../../../../../../../../etc/passwd will be normalized into /../../../../../../../etc/passwd, hence, a 400 bad request is returned.

But does that mean Nginx will protect Grafana against this kind of path traversal attack? May not be. It depends on how you configure the proxy_pass entry. Before I cover that, here is an example:

We can see that our path traversal still succeed and read the content of the sdk.ts, which is located two directories above the welcome plugin directory.

So here is first point, which is covered in the Nginx document

If proxy_pass is specified without a URI, the request URI is passed to the server in the same form as sent by a client when the original request is processed

Scroll back and examine my Nginx configuration, noticed that my proxy_pass entry is defined as http://localhost:3000, without a URI, hence, the original request will be forwarded to the Grafana backend. And in the first place, since my original URI is /public/plugins/welcome/../../sdk.ts, even after normalization by Nginx, it is /public/sdk.ts, which is a valid URI, hence, Nginx will not complain about it either.

This allows us to read arbitrary files up to three directory above the plugin directories. But the default plugin directories is deep at /usr/share/grafana/public/app/plugins/{plugin_id}, even being able to traverse up by 3 directories, there aren’t many files to read.

So here is the second point, which is covered in this post

URL consists of scheme:[//authority]path[?query][#fragment], and browsers don’t send #fragment. But how must a reverse proxy handle #fragment?
Nginx throws fragment off

So what would happen if my URI is /public/plugins/welcome/#/../../../../../../../../../etc/passwd?

Nginx will process until /public/plugins/welcome/, and forward the entire URI to the Grafana backend, and leads to path traversal all the way up to the root directory:

Attack Surface under Grafana

In this section, we try to examine the possible attack surface under Grafana when we are able to read files on the file system.

Grafana Database

We can try to read the database file, which is located at /var/lib/grafana/grafana.db by default, which is a sqlite database:

So, what is inside the Grafana database?

user table:

The password is hard to decrypt, using a slow hash algorithm with salt as defined in pkg/util/encoding.go:

// EncodePassword encodes a password using PBKDF2.
func EncodePassword(password string, salt string) (string, error) {
	newPasswd := pbkdf2.Key([]byte(password), []byte(salt), 10000, 50, sha256.New)
	return hex.EncodeToString(newPasswd), nil
}

user_auth_token:

It seems that user_auth_token is also stored as one-way hash, as defined in /pkg/services/auth/auth_token.go:

func hashToken(token string) string {
	hashBytes := sha256.Sum256([]byte(token + setting.SecretKey))
	return hex.EncodeToString(hashBytes[:])
}

I can’t think of a way to exploit this, if possible, please tell me =)

data_source:

The data_source tells Grafana where to pull the data from.

Finally there is something that we can exploit. In /pkg/cmd/grafana-cli/commands/datamigrations/encrypt_datasource_passwords_test.go:

func DecryptSecureJsonData(ds *models.DataSource) (map[string]string, error) {
	decrypted := make(map[string]string)
	for key, data := range ds.SecureJsonData {
		decryptedData, err := util.Decrypt(data, setting.SecretKey)
		if err != nil {
			return nil, err
		}

		decrypted[key] = string(decryptedData)
	}
	return decrypted, nil
}

util.Decrypt is defined in /pkg/util/encryption.go, you can re-use it to decrypt the encrypted password in the data source, or you can use the script here:

The secretkey is defined in the Grafana’s configuration file, which is located at /etc/grafana/grafana.ini, and it might contains other sensitive information as well. We will cover those next

Grafana Configuration File

Located at /etc/grafana/grafana.ini by default, which might contain several sensitive information. You can take a look at the default configuration file here and see what can be stored inside. I won’t go through them one by one.

grafana-image-renderer

Apart from the various credentials that could be leaked from the Grafana’s configuration file, another worth-mentioning entry is the grafana-image-renderer

[rendering]
# Options to configure a remote HTTP image rendering service, e.g. using https://github.com/grafana/grafana-image-renderer.
# URL to a remote HTTP image renderer service, e.g. http://localhost:8081/render, will enable Grafana to render panels and dashboards to PNG-images using HTTP requests to an external service.
server_url =
# If the remote HTTP image renderer service runs on a different server than the Grafana server you may have to configure this to a URL where Grafana is reachable, e.g. http://grafana.domain/.
callback_url =
# Concurrent render request limit affects when the /render HTTP endpoint is used. Rendering many images at the same time can overload the server,
# which this setting can help protect against by only allowing a certain amount of concurrent requests.
concurrent_render_request_limit = 30

grafana-image-renderer is a remote HTTP image rendering service, which you can ask the renderer to visit the Grafana panel, render into image and send to us.

The official guideline is to set up the grafana-image-renderer service inside a separate docker, and link it to the Grafana docker using Docker Compose like below:

version: '2'

services:
  grafana:
    image: grafana/grafana:latest
    ports:
      - '3000:3000'
    environment:
      GF_RENDERING_SERVER_URL: http://renderer:8081/render
      GF_RENDERING_CALLBACK_URL: http://grafana:3000/
      GF_LOG_FILTERS: rendering:debug
  renderer:
    image: grafana/grafana-image-renderer:latest
    ports:
      - 8081

Under this configuration, the renderer service is inaccessible from the Internet.

However, there is another option to run it as a standalone Node.js application.