Merge pull request #377 from hashicorp/alisdair/hclwrite-fuzz

hclwrite: Add fuzz testing

hclsyntax/fuzz/README.md

@@ -3,24 +3,49 @@
 This directory contains helper functions and corpuses that can be used to
 fuzz-test the `hclsyntax` parsers using [go-fuzz](https://github.com/dvyukov/go-fuzz).
 
-To fuzz, first install go-fuzz and its build tool in your `GOPATH`:
+## Work directory
+
+`go-fuzz` needs a working directory where it can keep state as it works.  This
+should ideally be in a ramdisk for efficiency, and should probably _not_ be on
+an SSD to avoid thrashing it. Here's how to create a ramdisk:
+
+### macOS
 
 ```
-$ make tools
+$ SIZE_IN_MB=1024
+$ DEVICE=`hdiutil attach -nobrowse -nomount ram://$(($SIZE_IN_MB*2048))`
+$ diskutil erasevolume HFS+ RamDisk $DEVICE
+$ export RAMDISK=/Volumes/RamDisk
+```
+
+### Linux
+
+```
+$ mkdir /mnt/ramdisk
+$ mount -t tmpfs -o size=1024M tmpfs /mnt/ramdisk
+$ export RAMDISK=/mnt/ramdisk
+```
+
+## Running the fuzzer
+
+Next, install `go-fuzz` and its build tool in your `GOPATH`:
+
+```
+$ make tools FUZZ_WORK_DIR=$RAMDISK
 ```
 
 Now you can fuzz one or all of the parsers:
 
 ```
-$ make fuzz-config FUZZ_WORK_DIR=/tmp/hcl2-fuzz-config
-$ make fuzz-expr  FUZZ_WORK_DIR=/tmp/hcl2-fuzz-expr
-$ make fuzz-template  FUZZ_WORK_DIR=/tmp/hcl2-fuzz-template
-$ make fuzz-traversal FUZZ_WORK_DIR=/tmp/hcl2-fuzz-traversal
+$ make fuzz-config FUZZ_WORK_DIR=$RAMDISK/hclsyntax-fuzz-config
+$ make fuzz-expr FUZZ_WORK_DIR=$RAMDISK/hclsyntax-fuzz-expr
+$ make fuzz-template FUZZ_WORK_DIR=$RAMDISK/hclsyntax-fuzz-template
+$ make fuzz-traversal FUZZ_WORK_DIR=$RAMDISK/hclsyntax-fuzz-traversal
 ```
 
-In all cases, set `FUZZ_WORK_DIR` to a directory where `go-fuzz` can keep state
-as it works. This should ideally be in a ramdisk for efficiency, and should
-probably _not_ be on an SSD to avoid thrashing it.
+~> Note: `go-fuzz` does not interact well with `goenv`. If you encounter build
+errors where the package `go.fuzz.main` could not be found, you may need to use
+a machine with a direct installation of Go.
 
 ## Understanding the result
 

hclwrite/fuzz/.gitignore

@@ -0,0 +1 @@
+fuzz*-fuzz.zip

hclwrite/fuzz/Makefile

@@ -0,0 +1,25 @@
+
+ifndef FUZZ_WORK_DIR
+$(error FUZZ_WORK_DIR is not set)
+endif
+
+default:
+	@echo "See README.md for usage instructions"
+
+fuzz-config: fuzz-exec-config
+
+fuzz-exec-%: fuzz%-fuzz.zip
+	go-fuzz -bin=./fuzz$*-fuzz.zip -workdir=$(FUZZ_WORK_DIR)
+
+fuzz%-fuzz.zip: %/fuzz.go
+	go-fuzz-build -x github.com/hashicorp/hcl/v2/hclwrite/fuzz/$*
+
+tools:
+	go get -u github.com/dvyukov/go-fuzz/go-fuzz
+	go get -u github.com/dvyukov/go-fuzz/go-fuzz-build
+
+clean:
+	rm fuzz*-fuzz.zip
+
+.PHONY: tools clean fuzz-config
+.PRECIOUS: fuzzconfig-fuzz.zip

hclwrite/fuzz/README.md

@@ -0,0 +1,82 @@
+# hclwrite fuzzing utilities
+
+This directory contains helper functions and corpuses that can be used to
+fuzz-test the `hclwrite` package using [go-fuzz](https://github.com/dvyukov/go-fuzz).
+
+## Work directory
+
+`go-fuzz` needs a working directory where it can keep state as it works.  This
+should ideally be in a ramdisk for efficiency, and should probably _not_ be on
+an SSD to avoid thrashing it. Here's how to create a ramdisk:
+
+### macOS
+
+```
+$ SIZE_IN_MB=1024
+$ DEVICE=`hdiutil attach -nobrowse -nomount ram://$(($SIZE_IN_MB*2048))`
+$ diskutil erasevolume HFS+ RamDisk $DEVICE
+$ export RAMDISK=/Volumes/RamDisk
+```
+
+### Linux
+
+```
+$ mkdir /mnt/ramdisk
+$ mount -t tmpfs -o size=1024M tmpfs /mnt/ramdisk
+$ export RAMDISK=/mnt/ramdisk
+```
+
+## Running the fuzzer
+
+Next, install `go-fuzz` and its build tool in your `GOPATH`:
+
+```
+$ make tools FUZZ_WORK_DIR=$RAMDISK
+```
+
+Now you can fuzz the parser:
+
+```
+$ make fuzz-config FUZZ_WORK_DIR=$RAMDISK/hclwrite-fuzz-config
+```
+
+~> Note: `go-fuzz` does not interact well with `goenv`. If you encounter build
+errors where the package `go.fuzz.main` could not be found, you may need to use
+a machine with a direct installation of Go.
+
+## Understanding the result
+
+A small number of subdirectories will be created in the work directory.
+
+If you let `go-fuzz` run for a few minutes (the more minutes the better) it
+may detect "crashers", which are inputs that caused the parser to panic. Details
+about these are written to `$FUZZ_WORK_DIR/crashers`:
+
+```
+$ ls /tmp/hcl2-fuzz-config/crashers
+7f5e9ec80c89da14b8b0b238ec88969f658f5a2d
+7f5e9ec80c89da14b8b0b238ec88969f658f5a2d.output
+7f5e9ec80c89da14b8b0b238ec88969f658f5a2d.quoted
+```
+
+The base file above (with no extension) is the input that caused a crash. The
+`.output` file contains the panic stack trace, which you can use as a clue to
+figure out what caused the crash.
+
+A good first step to fixing a detected crasher is to copy the failing input
+into one of the unit tests in the `hclwrite` package and see it crash there
+too. After that, it's easy to re-run the test as you try to fix it. The
+file with the `.quoted` extension contains a form of the input that is quoted
+in Go syntax for easy copy-paste into a test case, even if the input contains
+non-printable characters or other inconvenient symbols.
+
+## Rebuilding for new Upstream Code
+
+An archive file is created for `go-fuzz` to use on the first run of each
+of the above, as a `.zip` file created in this directory. If upstream code
+is changed these will need to be deleted to cause them to be rebuilt with
+the latest code:
+
+```
+$ make clean
+```

hclwrite/fuzz/config/corpus/attr-expr.hcl

@@ -0,0 +1 @@
+foo = upper(bar + baz[1])

hclwrite/fuzz/config/corpus/attr-literal.hcl

@@ -0,0 +1 @@
+foo = "bar"

hclwrite/fuzz/config/corpus/attr.hcl

@@ -0,0 +1 @@
+a = foo.bar

hclwrite/fuzz/config/corpus/block-attrs.hcl

@@ -0,0 +1,3 @@
+block {
+  foo = true
+}

hclwrite/fuzz/config/corpus/block-comment.hcl

@@ -0,0 +1,4 @@
+/* multi
+   line
+   comment
+*/

hclwrite/fuzz/config/corpus/block-empty.hcl

@@ -0,0 +1,2 @@
+block {
+}

hclwrite/fuzz/config/corpus/block-nested.hcl

@@ -0,0 +1,5 @@
+block {
+  another_block {
+    foo = bar
+  }
+}

hclwrite/fuzz/config/corpus/complex.hcl

@@ -0,0 +1 @@
+a = foo.bar[1].baz["foo"].pizza

hclwrite/fuzz/config/corpus/escape-dollar.hcl

@@ -0,0 +1 @@
+a = "hi $${var.foo}"

hclwrite/fuzz/config/corpus/escape-newline.hcl

@@ -0,0 +1 @@
+a = "bar\nbaz"

hclwrite/fuzz/config/corpus/function-call-tmpl.hcl

@@ -0,0 +1 @@
+a = "b ${title(var.name)}

hclwrite/fuzz/config/corpus/function-call.hcl

@@ -0,0 +1 @@
+a = title(var.name)

hclwrite/fuzz/config/corpus/hash-comment.hcl

@@ -0,0 +1 @@
+# another comment

hclwrite/fuzz/config/corpus/index.hcl

@@ -0,0 +1 @@
+a = foo[1]

hclwrite/fuzz/config/corpus/int-tmpl.hcl

@@ -0,0 +1 @@
+a = "foo ${42}"

hclwrite/fuzz/config/corpus/int.hcl

@@ -0,0 +1 @@
+a = 42

hclwrite/fuzz/config/corpus/just-interp.hcl

@@ -0,0 +1 @@
+a = "${var.bar}"

hclwrite/fuzz/config/corpus/literal.hcl

@@ -0,0 +1 @@
+a = foo

hclwrite/fuzz/config/corpus/lots-of-comments.hcl

@@ -0,0 +1,14 @@
+// comment
+block {
+  // another comment
+  another_block { # comment
+    // comment
+    foo = bar
+  }
+
+  /* commented out block
+  blah {
+    bar = foo
+  }
+  */
+}

hclwrite/fuzz/config/corpus/slash-comment.hcl

@@ -0,0 +1 @@
+// comment

hclwrite/fuzz/config/corpus/splat-attr.hcl

@@ -0,0 +1 @@
+a = foo.bar.*.baz

hclwrite/fuzz/config/corpus/splat-dot-full.hcl

@@ -0,0 +1 @@
+a = foo.bar.*

hclwrite/fuzz/config/corpus/splat-full.hcl

@@ -0,0 +1 @@
+a = foo.bar[*].baz

hclwrite/fuzz/config/corpus/traversal-dot-index-terminal.hcl

@@ -0,0 +1 @@
+a = foo.bar.0

hclwrite/fuzz/config/corpus/traversal-dot-index.hcl

@@ -0,0 +1 @@
+a = foo.bar.4.baz

hclwrite/fuzz/config/corpus/traversal-index.hcl

@@ -0,0 +1 @@
+a = foo.bar[4].baz

hclwrite/fuzz/config/corpus/utf8.hcl

@@ -0,0 +1 @@
+foo = "föo ${föo("föo")}"

hclwrite/fuzz/config/corpus/var.hcl

@@ -0,0 +1 @@
+a = var.bar

hclwrite/fuzz/config/fuzz.go

@@ -0,0 +1,24 @@
+package fuzzconfig
+
+import (
+	"io/ioutil"
+
+	"github.com/hashicorp/hcl/v2"
+	"github.com/hashicorp/hcl/v2/hclwrite"
+)
+
+func Fuzz(data []byte) int {
+	file, diags := hclwrite.ParseConfig(data, "<fuzz-conf>", hcl.Pos{Line: 1, Column: 1})
+
+	if diags.HasErrors() {
+		return 0
+	}
+
+	_, err := file.WriteTo(ioutil.Discard)
+
+	if err != nil {
+		return 0
+	}
+
+	return 1
+}

json/fuzz/README.md

@@ -3,21 +3,46 @@
 This directory contains helper functions and corpuses that can be used to
 fuzz-test the HCL JSON parser using [go-fuzz](https://github.com/dvyukov/go-fuzz).
 
-To fuzz, first install go-fuzz and its build tool in your `GOPATH`:
+## Work directory
+
+`go-fuzz` needs a working directory where it can keep state as it works.  This
+should ideally be in a ramdisk for efficiency, and should probably _not_ be on
+an SSD to avoid thrashing it. Here's how to create a ramdisk:
+
+### macOS
 
 ```
-$ make tools
+$ SIZE_IN_MB=1024
+$ DEVICE=`hdiutil attach -nobrowse -nomount ram://$(($SIZE_IN_MB*2048))`
+$ diskutil erasevolume HFS+ RamDisk $DEVICE
+$ export RAMDISK=/Volumes/RamDisk
 ```
 
-Now you can fuzz one or all of the parsers:
+### Linux
 
 ```
-$ make fuzz-config FUZZ_WORK_DIR=/tmp/hcl2-fuzz-config
+$ mkdir /mnt/ramdisk
+$ mount -t tmpfs -o size=1024M tmpfs /mnt/ramdisk
+$ export RAMDISK=/mnt/ramdisk
 ```
 
-In all cases, set `FUZZ_WORK_DIR` to a directory where `go-fuzz` can keep state
-as it works. This should ideally be in a ramdisk for efficiency, and should
-probably _not_ be on an SSD to avoid thrashing it.
+## Running the fuzzer
+
+Next, install `go-fuzz` and its build tool in your `GOPATH`:
+
+```
+$ make tools FUZZ_WORK_DIR=$RAMDISK
+```
+
+Now you can fuzz the parser:
+
+```
+$ make fuzz-config FUZZ_WORK_DIR=$RAMDISK/json-fuzz-config
+```
+
+~> Note: `go-fuzz` does not interact well with `goenv`. If you encounter build
+errors where the package `go.fuzz.main` could not be found, you may need to use
+a machine with a direct installation of Go.
 
 ## Understanding the result