Important note: A number of problems in this test suite have no antiderivative in closed form. This means the antiderivative of these integrals can not be expressed in terms of elementary, special functions or Hypergeometric2F1 functions. RootSum and RootOf are not allowed.
If a CAS returns the above integral unevaluated within the time limit, then the result is counted as passed and assigned an A grade.
However, if CAS times out, then it is assigned an F grade even if the integral is not integrable, as this implies CAS could not determine that the integral is not integrable in the time limit.
If a CAS returns an antiderivative to such an integral, it is assigned an A grade automatically and this special result is listed in the introduction section of each individual test report to make it easy to identify as this can be important result to investigate.
The results given in in the table below reflects the above.
| System | % solved | % Failed |
| Fricas | 99.92 ( 10327 ) | 0.08 ( 8 ) |
| Maple | 97.49 ( 10076 ) | 2.51 ( 259 ) |
| Mathematica | 97.37 ( 10063 ) | 2.63 ( 272 ) |
| Sympy | 95.43 ( 9863 ) | % 4.57 ( 472 ) |
| Maxima | 93.17 ( 9629 ) | 6.83 ( 706 ) |
| Mupad | 90.13 ( 9315 ) | 9.87 ( 1020 ) |
| Giac | 85.09 ( 8794 ) | 14.91 ( 1541 ) |
| Rubi | 63.38 ( 6550 ) | 36.62 ( 3785 ) |
The table below gives additional break down of the grading of quality of the antiderivatives generated by each CAS. The grading is given using the letters A,B,C and F with A being the best quality. The grading is accomplished by comparing the antiderivative generated with the optimal antiderivatives included in the test suite. The following table describes the meaning of these grades.
| grade |
description |
| A |
Integral was solved and antiderivative is optimal in quality and leaf size. |
| B |
Integral was solved and antiderivative is optimal in quality but leaf size is larger than twice the optimal antiderivatives leaf size. |
| C |
Integral was solved and antiderivative is non-optimal in quality. This can be due to one or more of the following reasons
|
| F |
Integral was not solved. Either the integral was returned unevaluated within the time limit, or it timed out, or CAS hanged or crashed or an exception was raised. |
Grading is implemented for all CAS systems. Based on the above, the following table summarizes the grading for this test suite.
| System | % A grade | % B grade | % C grade | % F grade |
| Mathematica | 88.51 | 7.80 | 0.81 | 2.63 |
| Fricas | 81.60 | 18.18 | 0.15 | 0.08 |
| Maple | 79.98 | 9.78 | 7.73 | 2.51 |
| Sympy | 77.10 | 18.16 | 0.17 | 4.57 |
| Maxima | 63.97 | 23.97 | 5.23 | 6.83 |
| Giac | 63.83 | 20.98 | 0.28 | 14.91 |
| Rubi | 49.55 | 12.19 | 1.64 | 36.62 |
| Mupad | N/A | 90.13 | 0.00 | 9.87 |
The following is a Bar chart illustration of the data in the above table.
The figure below compares the CAS systems for each grade level.
The following table shows the distribution of the different types of failure for each CAS. There are 3 types of reasons why it can fail. The first is when CAS returns back the input within the time limit, which means it could not solve it. This the typical normal failure F .
The second is due to time out. CAS could not solve the integral within the 3 minutes time limit which is assigned F(-1).
The third is due to an exception generated. Assigned F(-2). This most likely indicates an interface problem between sagemath and the CAS (applicable only to FriCAS, Maxima and Giac) or it could be an indication of an internal error in CAS. This type of error requires more investigations to determine the cause.
| System |
Number failed |
Percentage normal failure |
Percentage timeout failure |
Percentage exception failure |
| Rubi |
3785 |
95.32 % |
4.68 % |
0.00 % |
| Mathematica |
272 |
94.85 % |
5.15 % |
0.00 % |
| Maple | 259 | 96.53 % | 3.47 % | 0.00 % |
| Fricas | 8 | 0.00 % | 0.00 % | 100.00 % |
| Giac |
1541 |
68.33 % |
18.95 % |
12.72 % |
| Maxima |
706 |
83.99 % |
3.68 % |
12.32 % |
| Sympy |
472 |
1.27 % |
83.47 % |
15.25 % |
| Mupad |
1020 |
95.69 % |
4.31 % |
0.00 % |