How the ByteCTF Guess Cookie Challenge Was Designed

0x01 Preface

A while ago I created a misc challenge for the ByteCTF Masters competition. It grew out of my recent research into a middleware product, and I found the idea interesting enough to share the challenge-design process here.

0x02 A Small Joke

Message-oriented middleware has become a key component of distributed systems and microservice architectures. Common products include ActiveMQ, Kafka, RabbitMQ, and RocketMQ. RabbitMQ is widely used in finance, while Kafka is common in big-data systems, and over time RabbitMQ has become one of the most popular and widely deployed message queues. Yet security research on RabbitMQ and its exploitation remains relatively scarce. This challenge focuses on the security of RabbitMQ's Erlang-node communication and tests contestants' understanding of the full communication process.

After opening the supplied pcap file, we can see that the main protocols used between 172.20.10.24 and 172.20.10.4 include HTTP, TCP, AMQP, and ErlDP, as shown below:

The challenge includes fake flags. Searching directly for a flag finds fakerflag{ByteDance@2024} and this is a faker flag that is ByteCTF{1e1111dce-cdf7-423f-8a8b-a62dec323d17}—just a small joke.

0x03 Communication Flow Analysis

Now to the main topic. According to the challenge description:

Analyze this traffic and recover the cookie used for communication between 172.20.10.24 and 172.20.10.4.

According to the captured protocol data, the HTTP cookie is:

Converting it to a 32-character lowercase MD5 value produced an incorrect flag, so I examined the other protocols: AMQP, EPMD, and ErlDP.

These three protocols primarily support RabbitMQ. AMQP—the Advanced Message Queuing Protocol—is an open-standard application-layer protocol for asynchronous messaging in distributed systems, and is one of the principal protocols supported by brokers such as RabbitMQ. AMQP supports several authentication mechanisms, including the commonly used PLAIN SASL. With PLAIN SASL, a client authenticates by providing a username and password.

AMQP also supports other SASL mechanisms, such as EXTERNAL and ANONYMOUS. The mechanism used depends on the server configuration and security requirements. After authentication, the client must complete several steps before it can transfer messages, including establishing a connection, creating a channel, and declaring exchanges and queues. AMQP itself does not use cookies to manage sessions; it maintains communication state through long-lived connections and channels. We can therefore exclude it and focus on the two remaining protocols: EPMD and ErlDP.

These are the two protocols examined by the challenge—in other words, the Erlang node protocol's authentication process.

EPMD stands for Erlang Port Mapper Daemon. In RabbitMQ, it primarily acts as a name server, with the following main functions:

Maps symbolic node names to their actual IP addresses and port numbers.
Maintains a registry of active Erlang nodes.
Helps Erlang nodes establish their initial connection.

EPMD normally runs on port 4369 and serves nodes in an Erlang cluster.

The Erlang Distribution Protocol—sometimes shortened to Erlang Distribution or ErlDP—is the core protocol used by the Erlang language for communication in distributed systems. Its main features include:

Used for communication between Erlang nodes.
Supports remote procedure calls (RPC) and message passing.
Provides built-in fault tolerance and error handling.

For authentication between nodes, Erlang uses a mechanism called the "Magic Cookie":

Every Erlang node has a cookie, represented as a string.
When two nodes attempt to connect, they exchange and compare their respective cookies.
A connection is established only when the two nodes have the same cookie.
The cookie is normally stored in a file named .erlang.cookie.

Although this mechanism is called a "cookie," it is entirely different from the HTTP cookies used by web browsers. An Erlang cookie is a simple shared secret used to authenticate nodes. The final flag for this challenge is the 32-character lowercase MD5 value of that cookie. Filtering for protocols related to Erlang communication (epmd || erldp) reveals the following:

To understand this traffic, we first need to examine how RabbitMQ authenticates communication between Erlang nodes. The process is as follows:

Client Node Server Node | | | 1. SEND (name, flags, creation) | |----------------------------------------------->| | | | 2. CHALLENGE (challenge, flags) | |<-----------------------------------------------| | | | 3. CHALLENGE_REPLY (digest) | |----------------------------------------------->| | | | 4. CHALLENGE_ACK (digest) | |<-----------------------------------------------| | |

Now let us examine each step in detail. Step 1, SEND: the client node sends a SEND message containing the following information:

Node name: identifies the client node
Flags: contain version information and other metadata
Creation information: distinguishes nodes with the same name that were created at different times

{SEND, NodeName, % for example, 'rabbit@node01' Flags, % for example, [DFLAG_EXTENDED_REFERENCES, DFLAG_DIST_MONITOR, ...] Creation % for example, 1 }

Corresponds to lines 471, 517, 851, and 970 in the pcap file

Step 2, CHALLENGE: after receiving SEND, the server node responds with a CHALLENGE message containing:

Challenge: a randomly generated large integer
Flags: feature flags supported by the server

{CHALLENGE, Challenge, % for example, 1234567890 Flags % for example, [DFLAG_EXTENDED_REFERENCES, DFLAG_DIST_MONITOR, ...] }

Corresponds to lines 475, 521, 855, and 974 in the pcap file

Step 3, CHALLENGE_REPLY: after receiving the CHALLENGE, the client calculates and sends its response:

Digest: an MD5 hash calculated from the shared "magic cookie" and challenge

{CHALLENGE_REPLY, Digest % MD5 calculated by the server }

Corresponds to line 976 in the pcap file

Step 4—the final step—CHALLENGE_ACK: the server verifies the client's response and, if it is correct, sends an ACK:

Digest: a digest calculated by the server in the same way and used for mutual authentication

{CHALLENGE_ACK, Digest % MD5 calculated by the server }

Corresponds to line 977 in the pcap file:

That is the complete validation process. If the protocol flow and pcap still do not make the details clear, the Erlang node cookie-authentication code is another useful reference. The relevant definitions are in erlang/otp/lib/jinterface/java_src/com/ericsson/otp/erlang/AbstractConnection.java in the official repository. Its main code is shown below:

JAVA

    protected void recvChallengeAck(final int our_challenge)
            throws IOException, OtpAuthException {

        final byte[] her_digest = new byte[16];
        try {
            final byte[] buf = read2BytePackage();
            @SuppressWarnings("resource")
            final OtpInputStream ibuf = new OtpInputStream(buf, 0);
            final int tag = ibuf.read1();
            if (tag != ChallengeAck) {
                throw new IOException("Handshake protocol error");
            }
            ibuf.readN(her_digest);
            final byte[] our_digest = genDigest(our_challenge,
                    localNode.cookie());
            if (!digests_equals(her_digest, our_digest)) {
                throw new OtpAuthException("Peer authentication error.");
            }
        } catch (final OtpErlangDecodeException e) {
            throw new IOException("Handshake failed - not enough data");
        } catch (final Exception e) {
            throw new OtpAuthException("Peer authentication error.");
        }

        if (traceLevel >= handshakeThreshold) {
            System.out.println("<- " + "HANDSHAKE recvChallengeAck" + " from="
                    + peer.node + " digest=" + hex(her_digest) + " local="
                    + localNode);
        }
    }
...

    protected void sendChallengeReply(final int challenge, final byte[] digest)
            throws IOException {

        @SuppressWarnings("resource")
        final OtpOutputStream obuf = new OtpOutputStream();
        obuf.write2BE(21);
        obuf.write1(ChallengeReply);
        obuf.write4BE(challenge);
        obuf.write(digest);
        obuf.writeToAndFlush(socket.getOutputStream());

        if (traceLevel >= handshakeThreshold) {
            System.out.println("-> " + "HANDSHAKE sendChallengeReply"
                    + " challenge=" + challenge + " digest=" + hex(digest)
                    + " local=" + localNode);
        }
    }

...
    protected byte[] genDigest(final int challenge, final String cookie) {
        int i;
        long ch2;

        if (challenge < 0) {
            ch2 = 1L << 31;
            ch2 |= challenge & 0x7FFFFFFF;
        } else {
            ch2 = challenge;
        }
        final OtpMD5 context = new OtpMD5();
        context.update(cookie);
        context.update("" + ch2);

        final int[] tmp = context.final_bytes();
        final byte[] res = new byte[tmp.length];
        for (i = 0; i < tmp.length; ++i) {
            res[i] = (byte) (tmp[i] & 0xFF);
        }
        return res;
    }

The broad flow is as follows: Generate the digest: call genDigest() with three parameters—the challenge, the cookie, and a fixed string.

JAVA

protected byte[] genDigest(final int challenge, final String cookie) {
    // ...
    context.update(cookie);
    context.update("" + ch2);
    // ...
}

Exchange challenges: both sides exchange challenge values without directly exchanging the cookie.

Verify the digest: each side combines the received challenge with its own cookie to generate a digest, then compares it with the digest sent by the peer.

JAVA

final byte[] our_digest = genDigest(our_challenge, localNode.cookie());
if (!digests_equals(her_digest, our_digest)) {
    throw new OtpAuthException("Peer authentication error.");
}

Determine whether the cookie is correct: if both sides generate the same digest, the cookie is considered valid. Only peers using the same cookie can produce identical digests.

Set the cookieOk flag: when authentication succeeds, set cookieOk = true.

TEXT

java
Copy
cookieOk = true;
sendCookie = false;

Subsequent communication: once cookieOk is true, later messages do not repeat the full authentication process.

0x04 A Worked Example

Here is a concrete example. Remote IP: 101.x.x.145 The main flow is as follows:

Step	Message type	Direction	Contents	Description
1	`SEND_NAME`	Local → Remote	`rabbit@nodes`	The local node sends its node name to the remote node
2	`SEND_STATUS`	Remote → Local	`ok`	After finding the node, the remote node returns a confirmation status
3	`SEND_CHALLENGE`	Remote → Local	`0xd47f02d3`	The remote node sends a challenge to the local node
4	`SEND_CHALLENGE_REPLY`	Local → Remote	Challenge: `0x89164a06` Digest: `1fd30d5a0286c8bf72535e80db246256`	The local node replies with its own challenge and digest
5	`SEND_CHALLENGE_ACK`	Remote → Local	Digest: `aa8cbfa4f6401bac2aed672a11c8a7a8`	The remote node compares the received digest and, if it matches, returns its own digest

Sent by remote — challenge: 0xd47f02d3 Sent by remote — digest: aa8cbfa4f6401bac2aed672a11c8a7a8 Sent by local — challenge: 0x89164a06 Sent by local — digest: 1fd30d5a0286c8bf72535e80db246256 Remote cookie: VLWCCLWXYOZNQPJKAKIO Local cookie: VLWCCLWXYOZNQPJKAKIO 0x89164a06 and VLWCCLWXYOZNQPJKAKIO produce aa8cbfa4f6401bac2aed672a11c8a7a8 0xd47f02d3 and VLWCCLWXYOZNQPJKAKIO produce 1fd30d5a0286c8bf72535e80db246256

As shown below:

If we obtain the digest returned during SEND_CHALLENGE_ACK, we can easily guess the cookie when it was generated by default rather than customized by the user. The validation script for the preceding example is shown below:

JAVA

package Rabbitmq;

public class CookieHashDemo {
    public static byte[] genDigest(final int challenge, final String cookie) {
        int i;
        long ch2;

        if (challenge < 0) {
            ch2 = 1L << 31;
            ch2 |= challenge & 0x7FFFFFFF;
        } else {
            ch2 = challenge;
        }
        final OtpMD5 context = new OtpMD5();
        context.update(cookie);
        context.update("" + ch2);

        final int[] tmp = context.final_bytes();
        final byte[] res = new byte[tmp.length];
        for (i = 0; i < tmp.length; ++i) {
            res[i] = (byte) (tmp[i] & 0xFF);
        }
        return res;
    }
    private static boolean digests_equals(final byte[] a, final byte[] b) {
        int i;
        for (i = 0; i < 16; ++i) {
            if (a[i] != b[i]) {
                return false;
            }
        }
        return true;
    }
    static String hex0(final byte x) {
        final char tab[] = { '0', '1', '2', '3', '4', '5', '6', '7', '8', '9',
                'a', 'b', 'c', 'd', 'e', 'f' };
        int uint;
        if (x < 0) {
            uint = x & 0x7F;
            uint |= 1 << 7;
        } else {
            uint = x;
        }
        return "" + tab[uint >>> 4] + tab[uint & 0xF];
    }
    static String hex(final byte[] b) {
        final StringBuffer sb = new StringBuffer();
        try {
            int i;
            for (i = 0; i < b.length; ++i) {
                sb.append(hex0(b[i]));
            }
        } catch (final Exception e) {
            // Debug function, ignore errors.
        }
        return sb.toString();

    }
    public static void main(String[] args) {
        String local_cookie = "VLWCCLWXYOZNQPJKAKIO";
        int local_challenge = 0x89164a06;
        String remote_digest = "aa8cbfa4f6401bac2aed672a11c8a7a8";
        byte[] our_digest = genDigest(local_challenge,local_cookie);
        System.out.println(hex(our_digest));
        if (!digests_equals(remote_digest.getBytes(), hex(our_digest).getBytes())) {
            System.out.println("Peer authentication error.");
        }else {
            System.out.println("Success!");
        }
    }
}

0x05 Solution

Returning to the challenge, the pcap has already given us the challenge value (0x60ea7bde) and the digest returned during SEND_CHALLENGE_ACK (f0e2967976d3ad1d0e8d2e85e7146f1a). We only need to fuzz locally to infer the cookie.

There is one problem: before fuzzing, we must know what a generated cookie looks like and how it is produced; otherwise we cannot write the fuzzing script. In practice, it is easy to set up a RabbitMQ service on your own machine.

By default, RabbitMQ stores its cookie in /var/lib/rabbitmq/.erlang.cookie. It consists of 20 uppercase English letters, as in the following example:

The cookie-generation algorithm is shown below (official repository: /erlang/otp/blob/master/lib/kernel/src/auth.erl):

ERLANG

-module(cookie_generator).
-export([create_cookie/1]).

%% next_random/1 function
next_random(X) ->
    (X*17059465+1) band 16#fffffffff.
...
%% random_cookie/3 function
random_cookie(0, _, Result) ->
    lists:reverse(Result);
random_cookie(Count, X0, Result) ->
    X = next_random(X0),
    Letter = X*($Z-$A+1) div 16#1000000000 + $A,
    random_cookie(Count-1, X, [Letter|Result]).
...
%% create_cookie/1 function
create_cookie(Name) ->
    io:format("Seed_1: ~p~n", [abs(erlang:monotonic_time() bxor erlang:unique_integer())]),
    Seed = abs(erlang:monotonic_time() bxor erlang:unique_integer()),
    io:format("Seed_2: ~p~n", [abs(erlang:monotonic_time() bxor erlang:unique_integer())]),
    Cookie = random_cookie(20, Seed, []),
    io:format("Cookie: ~p~n", [Cookie]),
    Cookie.

The key value in this cookie-generation logic is the seed.

Seed = abs(erlang:monotonic_time() bxor erlang:unique_integer()),

erlang:monotonic_time(): the time, in nanoseconds, from when the Erlang VM started until the function is called
erlang:unique_integer(): returns an integer

In practice, these two values are difficult to predict.

However, XORing these two values produces a large integer.

We can therefore simulate cookie generation repeatedly—restarting the Erlang VM each time—and observe the characteristics of the seed:

Observation showed that most seeds fell between 350,000,000 and 550,000,000. Their magnitude also depends on the machine: systems with more processors tend to generate larger seeds. The following statistics show the ranges of 300,000 seeds generated on a single-core system:

Because the cracking is performed locally, it is actually quite fast. Once the approximate seed range is known, we can search from 300000000 to 900000000. Starting from 1 would also work, but would take longer. The final script is shown below: Payload.java

JAVA

package Rabbitmq;

public class Payload {
    public static byte[] genDigest(final int challenge, final String cookie) {
        int i;
        long ch2;

        if (challenge < 0) {
            ch2 = 1L << 31;
            ch2 |= challenge & 0x7FFFFFFF;
        } else {
            ch2 = challenge;
        }
        final OtpMD5 context = new OtpMD5();
        context.update(cookie);
        context.update("" + ch2);

        final int[] tmp = context.final_bytes();
        final byte[] res = new byte[tmp.length];
        for (i = 0; i < tmp.length; ++i) {
            res[i] = (byte) (tmp[i] & 0xFF);
        }
        return res;
    }
    private static boolean digests_equals(final byte[] a, final byte[] b) {
        int i;
        for (i = 0; i < 16; ++i) {
            if (a[i] != b[i]) {
                return false;
            }
        }
        return true;
    }
    static String hex0(final byte x) {
        final char tab[] = { '0', '1', '2', '3', '4', '5', '6', '7', '8', '9',
                'a', 'b', 'c', 'd', 'e', 'f' };
        int uint;
        if (x < 0) {
            uint = x & 0x7F;
            uint |= 1 << 7;
        } else {
            uint = x;
        }
        return "" + tab[uint >>> 4] + tab[uint & 0xF];
    }
    static String hex(final byte[] b) {
        final StringBuffer sb = new StringBuffer();
        try {
            int i;
            for (i = 0; i < b.length; ++i) {
                sb.append(hex0(b[i]));
            }
        } catch (final Exception e) {
        }
        return sb.toString();

    }
    public static long nextRandom(long x) {
        return (x * 17059465 + 1) & 0xfffffffffL;
    }

    public static String deriveCookie(long seed, int size) {
        long x = seed;
        char[] cookie = new char[size];
        for (int i = size - 1; i >= 0; i--) {
            x = nextRandom(x);
            cookie[i] = (char) ('A' + (26 * x) / 0x1000000000L);
        }
        return new String(cookie);
    }
    private static double nanoToMinutes(long nanos) {
        return nanos / (60.0 * 1_000_000_000);
    }
    public static void main(String[] args) {
        long seed_start = 300000000;
        long seed_end = 900000000;
        int size = 20;
        int local_challenge = 0x60ea7bde;
        String remote_digest = "f0e2967976d3ad1d0e8d2e85e7146f1a";
        long startTimeNano = System.nanoTime();
        for(long i = seed_start; i<=seed_end; i++){
            String local_cookie =  deriveCookie(i, size);
            byte[] our_digest = genDigest(local_challenge,local_cookie);
            if (!digests_equals(remote_digest.getBytes(), hex(our_digest).getBytes())) {
                System.out.println("[*] Seed: " + i + ", Try Cookie " + local_cookie + " : Peer authentication error.");
            }else {
                System.out.println("[*] Seed: " + i + ", Success! Your Cookie is " + local_cookie);
                long endTimeNano = System.nanoTime();
                long durationNano = endTimeNano - startTimeNano;
                double durationMinutes = nanoToMinutes(durationNano);
                System.out.printf("Fuzz time: %.4f minutes%n", durationMinutes);
                break;
            }
        }
    }
}

OtpMD5.java

JAVA

package Rabbitmq;
class OtpMD5 {

    static final long S11 = 7;
    static final long S12 = 12;
    static final long S13 = 17;
    static final long S14 = 22;
    static final long S21 = 5;
    static final long S22 = 9;
    static final long S23 = 14;
    static final long S24 = 20;
    static final long S31 = 4;
    static final long S32 = 11;
    static final long S33 = 16;
    static final long S34 = 23;
    static final long S41 = 6;
    static final long S42 = 10;
    static final long S43 = 15;
    static final long S44 = 21;

    private final long state[] = { 0x67452301L, 0xefcdab89L, 0x98badcfeL,
            0x10325476L };
    private final long count[] = { 0L, 0L };
    private final int buffer[];

    public OtpMD5() {
        buffer = new int[64];
        int i;
        for (i = 0; i < 64; ++i) {
            buffer[i] = 0;
        }
    }

    private int[] to_bytes(final String s) {
        final char tmp[] = s.toCharArray();
        final int ret[] = new int[tmp.length];
        int i;

        for (i = 0; i < tmp.length; ++i) {
            ret[i] = tmp[i] & 0xFF;
        }
        return ret;
    }

    private int[] clean_bytes(final int bytes[]) {
        final int ret[] = new int[bytes.length];
        int i;

        for (i = 0; i < bytes.length; ++i) {
            ret[i] = bytes[i] & 0xFF;
        }
        return ret;
    }

    private long shl(final long what, final int steps) {
        return what << steps & 0xFFFFFFFFL;
    }

    private long shr(final long what, final int steps) {
        return what >>> steps;
    }

    private long plus(final long a, final long b) {
        return a + b & 0xFFFFFFFFL;
    }

    private long not(final long x) {
        return ~x & 0xFFFFFFFFL;
    }

    private void to_buffer(final int to_start, final int[] from,
                           final int from_start, final int num) {
        int ix = num;
        int to_ix = to_start;
        int from_ix = from_start;
        while (ix-- > 0) {
            buffer[to_ix++] = from[from_ix++];
        }
    }

    private void do_update(final int bytes[]) {
        int index = (int) (count[0] >>> 3 & 0x3F);
        final long inlen = bytes.length;
        final long addcount = shl(inlen, 3);
        final long partlen = 64 - index;
        int i;

        count[0] = plus(count[0], addcount);

        if (count[0] < addcount) {
            ++count[1];
        }

        count[1] = plus(count[1], shr(inlen, 29));
        if (inlen >= partlen) {
            to_buffer(index, bytes, 0, (int) partlen);
            transform(buffer, 0);

            for (i = (int) partlen; i + 63 < inlen; i += 64) {
                transform(bytes, i);
            }

            index = 0;
        } else {
            i = 0;
        }
        to_buffer(index, bytes, i, (int) inlen - i);
    }

    @SuppressWarnings("unused")
    private void dumpstate() {
        System.out.println("state = {" + state[0] + ", " + state[1] + ", "
                + state[2] + ", " + state[3] + "}");
        System.out.println("count = {" + count[0] + ", " + count[1] + "}");
        System.out.print("buffer = {");
        int i;
        for (i = 0; i < 64; ++i) {
            if (i > 0) {
                System.out.print(", ");
            }
            System.out.print(buffer[i]);
        }
        System.out.println("}");
    }

    private long F(final long x, final long y, final long z) {
        return x & y | not(x) & z;
    }

    private long G(final long x, final long y, final long z) {
        return x & z | y & not(z);
    }

    private long H(final long x, final long y, final long z) {
        return x ^ y ^ z;
    }

    private long I(final long x, final long y, final long z) {
        return y ^ (x | not(z));
    }

    private long ROTATE_LEFT(final long x, final long n) {
        return shl(x, (int) n) | shr(x, (int) (32 - n));
    }

    private long FF(final long a, final long b, final long c, final long d,
                    final long x, final long s, final long ac) {
        long tmp = plus(a, plus(plus(F(b, c, d), x), ac));
        tmp = ROTATE_LEFT(tmp, s);
        return plus(tmp, b);
    }

    private long GG(final long a, final long b, final long c, final long d,
                    final long x, final long s, final long ac) {
        long tmp = plus(a, plus(plus(G(b, c, d), x), ac));
        tmp = ROTATE_LEFT(tmp, s);
        return plus(tmp, b);
    }

    private long HH(final long a, final long b, final long c, final long d,
                    final long x, final long s, final long ac) {
        long tmp = plus(a, plus(plus(H(b, c, d), x), ac));
        tmp = ROTATE_LEFT(tmp, s);
        return plus(tmp, b);
    }

    private long II(final long a, final long b, final long c, final long d,
                    final long x, final long s, final long ac) {
        long tmp = plus(a, plus(plus(I(b, c, d), x), ac));
        tmp = ROTATE_LEFT(tmp, s);
        return plus(tmp, b);
    }

    private void decode(final long output[], final int input[],
                        final int in_from, final int len) {
        int i, j;

        for (i = 0, j = 0; j < len; i++, j += 4) {
            output[i] = input[j + in_from] | shl(input[j + in_from + 1], 8)
                    | shl(input[j + in_from + 2], 16)
                    | shl(input[j + in_from + 3], 24);
        }
    }

    private void transform(final int block[], final int from) {
        long a = state[0];
        long b = state[1];
        long c = state[2];
        long d = state[3];
        final long x[] = new long[16];

        decode(x, block, from, 64);

        a = FF(a, b, c, d, x[0], S11, 0xd76aa478L);
        d = FF(d, a, b, c, x[1], S12, 0xe8c7b756L);
        c = FF(c, d, a, b, x[2], S13, 0x242070dbL);
        b = FF(b, c, d, a, x[3], S14, 0xc1bdceeeL);
        a = FF(a, b, c, d, x[4], S11, 0xf57c0fafL);
        d = FF(d, a, b, c, x[5], S12, 0x4787c62aL);
        c = FF(c, d, a, b, x[6], S13, 0xa8304613L);
        b = FF(b, c, d, a, x[7], S14, 0xfd469501L);
        a = FF(a, b, c, d, x[8], S11, 0x698098d8L);
        d = FF(d, a, b, c, x[9], S12, 0x8b44f7afL);
        c = FF(c, d, a, b, x[10], S13, 0xffff5bb1L);
        b = FF(b, c, d, a, x[11], S14, 0x895cd7beL);
        a = FF(a, b, c, d, x[12], S11, 0x6b901122L);
        d = FF(d, a, b, c, x[13], S12, 0xfd987193L);
        c = FF(c, d, a, b, x[14], S13, 0xa679438eL);
        b = FF(b, c, d, a, x[15], S14, 0x49b40821L);
        a = GG(a, b, c, d, x[1], S21, 0xf61e2562L);
        d = GG(d, a, b, c, x[6], S22, 0xc040b340L);
        c = GG(c, d, a, b, x[11], S23, 0x265e5a51L);
        b = GG(b, c, d, a, x[0], S24, 0xe9b6c7aaL);
        a = GG(a, b, c, d, x[5], S21, 0xd62f105dL);
        d = GG(d, a, b, c, x[10], S22, 0x2441453L);
        c = GG(c, d, a, b, x[15], S23, 0xd8a1e681L);
        b = GG(b, c, d, a, x[4], S24, 0xe7d3fbc8L);
        a = GG(a, b, c, d, x[9], S21, 0x21e1cde6L);
        d = GG(d, a, b, c, x[14], S22, 0xc33707d6L);
        c = GG(c, d, a, b, x[3], S23, 0xf4d50d87L);
        b = GG(b, c, d, a, x[8], S24, 0x455a14edL);
        a = GG(a, b, c, d, x[13], S21, 0xa9e3e905L);
        d = GG(d, a, b, c, x[2], S22, 0xfcefa3f8L);
        c = GG(c, d, a, b, x[7], S23, 0x676f02d9L);
        b = GG(b, c, d, a, x[12], S24, 0x8d2a4c8aL);
        a = HH(a, b, c, d, x[5], S31, 0xfffa3942L);
        d = HH(d, a, b, c, x[8], S32, 0x8771f681L);
        c = HH(c, d, a, b, x[11], S33, 0x6d9d6122L);
        b = HH(b, c, d, a, x[14], S34, 0xfde5380cL);
        a = HH(a, b, c, d, x[1], S31, 0xa4beea44L);
        d = HH(d, a, b, c, x[4], S32, 0x4bdecfa9L);
        c = HH(c, d, a, b, x[7], S33, 0xf6bb4b60L);
        b = HH(b, c, d, a, x[10], S34, 0xbebfbc70L);
        a = HH(a, b, c, d, x[13], S31, 0x289b7ec6L);
        d = HH(d, a, b, c, x[0], S32, 0xeaa127faL);
        c = HH(c, d, a, b, x[3], S33, 0xd4ef3085L);
        b = HH(b, c, d, a, x[6], S34, 0x4881d05L);
        a = HH(a, b, c, d, x[9], S31, 0xd9d4d039L);
        d = HH(d, a, b, c, x[12], S32, 0xe6db99e5L);
        c = HH(c, d, a, b, x[15], S33, 0x1fa27cf8L);
        b = HH(b, c, d, a, x[2], S34, 0xc4ac5665L);
        a = II(a, b, c, d, x[0], S41, 0xf4292244L);
        d = II(d, a, b, c, x[7], S42, 0x432aff97L);
        c = II(c, d, a, b, x[14], S43, 0xab9423a7L);
        b = II(b, c, d, a, x[5], S44, 0xfc93a039L);
        a = II(a, b, c, d, x[12], S41, 0x655b59c3L);
        d = II(d, a, b, c, x[3], S42, 0x8f0ccc92L);
        c = II(c, d, a, b, x[10], S43, 0xffeff47dL);
        b = II(b, c, d, a, x[1], S44, 0x85845dd1L);
        a = II(a, b, c, d, x[8], S41, 0x6fa87e4fL);
        d = II(d, a, b, c, x[15], S42, 0xfe2ce6e0L);
        c = II(c, d, a, b, x[6], S43, 0xa3014314L);
        b = II(b, c, d, a, x[13], S44, 0x4e0811a1L);
        a = II(a, b, c, d, x[4], S41, 0xf7537e82L);
        d = II(d, a, b, c, x[11], S42, 0xbd3af235L);
        c = II(c, d, a, b, x[2], S43, 0x2ad7d2bbL);
        b = II(b, c, d, a, x[9], S44, 0xeb86d391L);

        state[0] = plus(state[0], a);
        state[1] = plus(state[1], b);
        state[2] = plus(state[2], c);
        state[3] = plus(state[3], d);
    }

    public void update(final int bytes[]) {
        do_update(clean_bytes(bytes));
    }

    public void update(final String s) {
        do_update(to_bytes(s));
    }

    private int[] encode(final long[] input, final int len) {
        final int output[] = new int[len];
        int i, j;
        for (i = 0, j = 0; j < len; i++, j += 4) {
            output[j] = (int) (input[i] & 0xff);
            output[j + 1] = (int) (input[i] >>> 8 & 0xff);
            output[j + 2] = (int) (input[i] >>> 16 & 0xff);
            output[j + 3] = (int) (input[i] >>> 24 & 0xff);
        }
        return output;
    }

    public int[] final_bytes() {
        final int bits[] = encode(count, 8);
        int index, padlen;
        int padding[], i;
        int[] digest;

        index = (int) (count[0] >>> 3 & 0x3f);
        padlen = index < 56 ? 56 - index : 120 - index;
        padding = new int[padlen];
        padding[0] = 0x80;
        for (i = 1; i < padlen; ++i) {
            padding[i] = 0;
        }

        do_update(padding);

        do_update(bits);

        digest = encode(state, 16);

        return digest;
    }
}

This produces the following final cookie and seed values:

[*] Seed: 426271377, Success! Your Cookie is OXZHUNYQHJBWDLCGNUKZ

Single-threaded fuzzing time: 18.1437 minutes

After obtaining the cookie, convert it to a 32-character lowercase MD5 value:

The final flag is therefore: ByteCTF{e1347d3b4a848e4fc850b069dbfab71d}

0x06 Closing Notes

This was originally intended to be a web challenge, but for various reasons it was simplified into a misc challenge. I may discuss the rest another time.