Stress Testing Solutions For Comeptitive Programming
Not being able to catch corner cases is a problem that I faced while solving problems in contests or while doing practice. Sometimes the code that I wrote was correct for all the test cases that I had but the judge still gave me WA, which made me frustated and un-motivated to solve more problems.
To stress test my solutions I use shell scripts that automate pretty much all of the stuff for me.
To stress test you need 3 files and make sure these all are in the same directory -
- Your code (The code that you want to stress test).
- The correct code (This is the file against which your code will be stress tested). Now you may be wondering if I already have the holy grail of correct code why won’t I just submit that and be done with the problem. But either you
- Have written an optimised code with a better complexity. But you can’t figure out where it’s wrong. So writing a brute force and testing for smaller inputs might be helpful here.
- Are doing practice and you already have a couple of solutions by other peole which has been judged correctly. So you can use their code to generate inputs where your code fails.
- A program to generate random test cases.
Create this file with any name you want. In this directory use
g++ -o a.out filename.cpp to create the
Create this file with any name but preferably stress.cpp. Use
g++ -o stress.out stress.cpp to create the
Program to generate test cases
I am using python since it’s easier to quickly generate test cases of any kind. Keep the name of this python file as
Save the snippet below as file
# stress.sh #!/bin/bash start=0 curr_dir=$PWD while [ $start -lt 1000 ] do (( start++ )) if [ -e stress.py ] then python3 "$PWD/stress.py" > "$PWD/stressInput.txt" ./a.out < "$PWD/stressInput.txt" > "$PWD/myStressOutput.txt" ./stress.out < "$PWD/stressInput.txt" > "$PWD/stressOutput.txt" diff "$PWD/stressOutput.txt" "$PWD/myStressOutput.txt" || exit 1 echo -e "\033[0;32mDone for $start \xE2\x9C\x93 \033[0m" else echo -e "\033[0;31m Python file not found \xE2\x9D\x8C \033[0m" break fi done
The shell script first looks for a file with the name
stress.py. If it exists, we create test cases using this file and generate
stressOutput corresponding to the correct code and
myStressOutput.txt for the code we want to test.
diff is used to detect differences between the two files and if any, the process is stopped. You can then look at
stressInput.txt to see a corner test case and
myStressOutput.txt for your output and
stressOutput.txt for the correct output. And using this information you might be able to finally get a hang of where your code went wrong.
Now I have limited the number of test cases in each loop to 1000. You can increase that in the shell script or run it again if you want to continue the testing.
stress.sh in the directory where these 3 files are saved to run the testing.
If you are on a UNIX based system (like MacOS or Linux) you can follow these steps to make
stress.sh an executable and then run it from any directory in the system. This will save you the hassle of creating a new shell script file in every directory where you want to stress test your solutions.
- Make the shell scipt executable using
chmod +x stress.sh
- Copy the script to
cp stress.sh /usr/local/bin/stress.sh
I hope this post was useful. In case of any doubts, hit me up on Twitter at this thread.