Automated tests¶

Evaluation of weekly exercises uses automatic tests. They have the benefit that you can immediately know, if your program passed the tests or not, and how many points did you earn.

The drawback of automatic tests is that prints shown in the assignments must be exactly correct. Moreover, the error messages given by the automatic tests can be difficult to interpret. Therefore, this section provides you a list of points that you should check if you are wondering the feedback given by the automatic tests.

• Compare carefully the differences, which the automatic testing system shows between your output and the model output. Even a small difference (upper-case/lower-case letter, extra/missing empty space) is meaningful for the testing system.

• Check if there are files or directories generated by the compiler in your central repository (directories beginning with build, files with the suffix .o etc.). If there are such files/directories, remove them, since they may prevent the compiler of the tester working correctly.

  • By no means remove the files via web user interface, but remove them first locally in your own directory, and then push the change into your central repository.

  • To prevent the problem described above in the future, it is highly recommended to create a .gitignore file (theory and exercise).

• Uninitialized variables typically produce warnings or errors in program execution that are difficult to interpret. Therefore, please check that you have initialized all the variables. Remember also member variables, i.e. attributes of a class.

• If an automated test shows “Submission accepted” and the status is “In grading”, you should wait for a while and update the page if needed. If this situation lasts a long time (more than an hour) and updating the page has no effect, contact prog2@tuni.fi.

• If an automatic test gives no feedback, i.e. it gets stuck, you should suspect an infinite loop in your program. (The stuck can sometimes be due to a server trouble, but in such case, resubmitting after a while is typically effective.)

• If an automated test gives the following error message:

  Your program ran for over 90 seconds and it was killed. Usually this means that the program's execution got stuck in an endless loop or recursion. The program was killed during the last available io test below.

  it can mean an endless loop or recursion, but it can alternatively mean that your program crashed. A very usual reason for crashing is over-indexing a vector or an array. (This is easier to reveal, if you use at function instead of brackets.

  Note also that especially Windows computer can pass aforementioned errors (as well as other errors, too), even if automated tests fail. To better see how your program works, you should try to run it in Linux remote desktop.

• If your program has errors preventing the program to be run, the program (of course) cannot pass automated tests. Even if your program does not have such errors, it can still produce warnings. Warnings do not prevent the program to be run, so it can seem to work correctly. However, automated tests do not accept programs that produce warnings. In such case, the tester tells the reason for the rejection.

• If an automatic test gives the warning:

  control reaches end of non-void function...
  

  it means that a function returning a value is lacking a return statement, or the return statement is inside an if structure such that the compiler gives a warning about it.

• Some (especially large) assignments use unit tests. They are tests for checking if a single function works correctly. For them, it is not enough that the program as a whole works correctly, but each function must work correctly.

  • If working of a function is based, for example, on a return value of another function, a unit test may fail due to the other function, not the one, about which the automatic test complains.

  • A unit test may fail, if a function prints something that it should not print, i.e. the program seems to work correctly, but printing happens in a wrong place.