Create a flaky test

Question

Create a piece of code and a genuinely-looking test for it. However, the test must be flaky: when run multiple times, it must sometimes fail, sometimes succeed.

Be creative and try to hide the flakiness as much as possible. The most popular answer in (at least) 10 days wins.

Answer 1

JavaScript

Here is a very simple function that divides two given numbers:

function div(x, y) {
    return x / y;
}

And a test function that for 100 test cycles generates two random numbers from 0 to 10 and checks if the function returns the same result of division as provided in the test:

function test() {
    for (var i = 0; i < 100; i++) {
        var x = Math.floor(Math.random() * 10),
            y = Math.floor(Math.random() * 10);

        if (x / y === div(x, y)) {
            console.log('Test OK');
        } else {
            console.log('Test FAILED');
            break;
        }
    }
}

This example doesn't relate to the problem of math with floating numbers or any type casting tricks, as you may probably decide at first sight. Test will fail when x and y equal to 0. In this case JavaScript will always return NaN, which itself never equals to NaN.

Answer 2

C#, Thread-safe?

OK, let's write a kind of Dictionary that supports increasing and summing of values. Oh, and it's thread-safe. It even says so in its name:

public class ThreadSafeCounter<TKey>
{
    private Dictionary<TKey, int> _dictionary = new Dictionary<TKey, int>();
    private object _lock = new object();

    public bool ContainsKey(TKey key)
    {
        lock (_lock) { return _dictionary.ContainsKey(key); }
    }

    public void Add(TKey key, int value)
    {
        lock (_lock) { _dictionary[key] = value; }
    }

    public void Inc(TKey key)
    {
        lock (_lock) { _dictionary[key]++; }
    }

    public int GetSum()
    {
        lock (_lock) { return _dictionary.Values.Sum(); }
    }
}

So, since it's obviously thread-safe, let's fill it using several threads. What could possibly go wrong?

private void FillCounter(ThreadSafeCounter<int> counter)
{
    for (int i = 0; i < 10; i++)
    {
        if (counter.ContainsKey(i))
        {
            counter.Inc(i);
        }
        else
        {
            counter.Add(i, 1);
        }
    }
}

public void Test()
{
    var counter = new ThreadSafeCounter<int>();

    var thread = new Thread(() => FillCounter(counter));
    thread.Start();
    FillCounter(counter);
    thread.Join();

    int sum = counter.GetSum();
    if (sum != 20)
    {
        Console.WriteLine("What? Test failed!");
    }
}

If you know a bit about multi-threading, the problem in the test should be boringly obvious (Is that a word? "boringly"). But it might fool novices.

While each operation of ThreadSafeCounter is indeed thread-safe, the way the counter is used is not. Between checking for the existence of the key and adding it, the other thread might do the same. So, both threads end up setting the counter to 1 and none actually increases it. A classical race condition.

Answer 3

public bool IsLeapYear(int year)
{
    return year % 4 == 0 && year % 100 != 0 || year % 400 == 0;
}

[Test]
public void TestIsLeapYear()
{
    Assert.Equals(DateTime.IsLeapYear(DateTime.Now.Year), IsLeapYear(DateTime.Now.Year));
}

The logic is sound, but the test isn't. Run the test a few times at 2015.12.31. 23:59:59. The error is in getting the current time twice. It will fail about twice every four years (at the start and end of leap years, when there is a transition), so it's very well hidden.

Answer 4

I'm sure there are even better examples in PHP, but here's one:

<?php
// Computes the sum of two numbers
function sum($a, $b) {
    return $a + $b;
}

function test_add_numbers() {
    // Perform 100 tests
    for ($i = 0; $i < 100; $i++) {
        // Get two numbers between 0.1 and 0.9
        $a = rand(1, 9) / 10;
        $b = rand(1, 9) / 10;

        // Compute the sum of the two numbers: 0.2 .. 1.8
        $sum = $a + $b;

        // Make it integer: 2 .. 18
        $sum = $sum * 10;

        if (sum($a * 10, $b * 10) == (int)$sum) {
            print "Test OK\n";
        }
        else {
            print "Test FAILED\n";
            die;
        }
    }
}

test_add_numbers();

Click here for DEMO.

This exploits the fact that (int)((0.1 + 0.7)*10) evaluates to 7 (rather than 8 as expected). More details here.