The software development community is full of memes.
We replaced our monolith with micro services so that every outage could be more like a murder mystery.— Honest Status Page (@honest_update) October 7, 2015
I don’t mean LOLCATS or what you’d find on Reddit’s front-page, but instead “an idea, behavior, or style that spreads from person to person within a culture” (from Wikipedia).
Hey, that’s actually an interesting idea. Let’s do it.
Byte-Monkey takes inspiration from Netflix’s chaos-monkeys, which have become synonymous with fault-tolerance experiments through controlled failure injection. It runs on the JVM, and twiddles the bytecode of your app to introduce the kind of failures you might encounter such as exceptions and latency.
Using with a JVM app
Byte-Monkey is loaded as a java agent during JVM startup.
java -javaagent:byte-monkey.jar=mode:fault,rate:0.5,filter:org/eclipse/ -jar your-app.jar
The above configuration would run it in Fault mode (throw exceptions) but only instrument classes in packages under
org/eclipse/ and throw exceptions 50% of the time. More information about configuration options is on the GitHub Page
Why would I want to use this?
I've made a couple of posts previously about the utility of running controlled experiments to discover failure cases, at the application level or infrastructure level. Such drills give us the power and knowledge to answer questions about the behaviour of our multi-actor systems under different modes of failure.
Byte-Monkey addresses the scenario where you want to test something inside your JVM app that may not be triggered by external factors. It lets you answer questions like "If our db connection driver started exhibiting faults every 1/10 operations, does the app release connections properly or does it cause issues?" or "What happens if our http client suddenly starts holding onto connections for an extra 100ms?"
With an app like Byte-Monkey, you can turn up the chance of failures occurring and see how your system behaves without having to make any adjustments to the application code itself.
Byte-Monkey uses the JVM Instrumentation API. Implementing the API enables you to register transformers that can iterate over (and transform) class files as they are loaded by the JVM.
Each class is provided to the transformer as
byte. It’s best to use a library to take care of the abstraction between the JVM bytecode and its serialised format. Byte-Monkey uses Objectweb ASM which comes packaged with the JDK.
Byte-Monkey behaves roughly like the pseudocode below
for class in loaded-classfiles: for method in class: meddle-with-bytecode(method)
meddle-with-bytecode can either cause exceptions to be thrown or the thread to pause at the start of the method.
The bytecode of a simple "Hello, World!" method prior to having an exception injected looks like this:
public void printSomething() throws java.io.IOException; Code: 0: getstatic #2 // Field java/lang/System.out:Ljava/io/PrintStream; 3: ldc #3 // String Hello! 5: invokevirtual #4 // Method java/io/PrintStream.println:(Ljava/lang/String;)V 8: return
After being meddled with by the Byte-Monkey, it instead looks like this:
public void printSomething() throws java.io.IOException; Code: 0: invokestatic #20 // Method uk/co/probablyfine/bytemonkey/ByteMonkeyClassTransformer.shouldActivate:()Z 3: ifeq 16 6: new #22 // class uk/co/probablyfine/bytemonkey/ByteMonkeyException 9: dup 10: ldc #23 // String java/io/IOException 12: invokespecial #26 // Method uk/co/probablyfine/bytemonkey/ByteMonkeyException."<init>":(Ljava/lang/String;)V 15: athrow 16: getstatic #32 // Field java/lang/System.out:Ljava/io/PrintStream; 19: ldc #34 // String Hello! 21: invokevirtual #39 // Method java/io/PrintStream.println:(Ljava/lang/String;)V 24: return
This is the core of how Byte-Monkey works:
- 0: A call to
ByteMonkeyClassTransformer.shouldActivatewhich returns true/false.
- 3: If
shouldActivatewas false, we jump straight to instruction 16 - the beginning of the original code.
- 6: Create a new ByteMonkeyException, a subclass of RuntimeException that takes an exception name as an argument
- 10: Load the name of the exception that would be thrown (the first exception in the method signature, here an IOException)
- 12: Call the constructor with the IOException name
- 15: Here we go! Throw the exception
When the exception branch is activated, the output would resemble the following (taken from a failing test)
uk.co.probablyfine.bytemonkey.ByteMonkeyException: You've made a monkey out of me! Simulating throw of [java/io/IOException] at some.package.Class.methodName
The other mode, Latency, behaves in the same way but instead of creating and throwing exceptions it executes a
Currently, Byte-Monkey uses its own wrapper class to throw Exceptions since the type signature of constructors on most Exceptions aren't consistent. A scope for improvement would be building an actual instance of the exception in the type signature to activate legitimate try-catch blocks further up the chain.