Chapter 1
Introduction

This report gives the result of running the computer algebra independent integration problems.

The listing of the problems used by this report are

1. MathematicaSyntaxTestFiles.zip
2. MapleSyntaxTestFiles.zip

The above zip files were downloaded from rulebasedintegration.org.

The current number of problems in this test suite is [71994].

1.1 Listing of CAS systems tested

The following systems were tested at this time.

1. Mathematica 12.1 (64 bit) on windows 10.
2. Rubi 4.16.1 in Mathematica 12 on windows 10.
3. Maple 2020 (64 bit) on windows 10.
4. Maxima 5.43 on Linux. (via sagemath 8.9)
5. Fricas 1.3.6 on Linux (via sagemath 9.0)
6. Sympy 1.5 under Python 3.7.3 using Anaconda distribution.
7. Giac/Xcas 1.5 on Linux. (via sagemath 8.9)

Maxima, Fricas and Giac/Xcas were called from inside SageMath. This was done using SageMath integrate command by changing the name of the algorithm to use the different CAS systems.

Sympy was called directly using Python.

1.2 Results

Important note: A number of problems in this test suite have no antiderivative in closed form. This means the antiderivative of these integrals can not be expressed in terms of elementary, special functions or Hypergeometric2F1 functions. RootSum and RootOf are not allowed.

If a CAS returns the above integral unevaluated within the time limit, then the result is counted as passed and assigned an A grade.

However, if CAS times out, then it is assigned an F grade even if the integral is not integrable, as this implies CAS could not determine that the integral is not integrable in the time limit.

If a CAS returns an antiderivative to such an integral, it is assigned an A grade automatically and this special result is listed in the introduction section of each individual test report to make it easy to identify as this can be important result to investigate.

The results given in in the table below reflects the above.

System	solved	Failed
Rubi	% 99.52 ( 71651 )	% 0.48 ( 343 )
Mathematica	% 98.35 ( 70804 )	% 1.65 ( 1190 )
Maple	% 83.46 ( 60084 )	% 16.54 ( 11910 )
Fricas	% 68.07 ( 49005 )	% 31.93 ( 22989 )
Giac	% 52.51 ( 37804 )	% 47.49 ( 34190 )
Maxima	% 43.03 ( 30982 )	% 56.97 ( 41012 )
Sympy	% 32.41 ( 23332 )	% 67.59 ( 48662 )

Table 1.1:Percentage solved for each CAS

The table below gives additional break down of the grading of quality of the antiderivatives generated by each CAS. The grading is given using the letters A,B,C and F with A being the best quality. The grading is accomplished by comparing the antiderivative generated with the optimal antiderivatives included in the test suite. The following table describes the meaning of these grades.

grade	description
A	Integral was solved and antiderivative is optimal in quality and leaf size.
B	Integral was solved and antiderivative is optimal in quality but leaf size is larger than twice the optimal antiderivatives leaf size.
C	Integral was solved and antiderivative is non-optimal in quality. This can be due to one or more of the following reasons antiderivative contains a hypergeometric function and the optimal antiderivative does not. antiderivative contains a special function and the optimal antiderivative does not. antiderivative contains the imaginary unit and the optimal antiderivative does not.
F	Integral was not solved. Either the integral was returned unevaluated within the time limit, or it timed out, or CAS hanged or crashed or an exception was raised.

Table 1.2:Description of grading applied to integration result

Grading is implemented for all CAS systems. Based on the above, the following table summarizes the grading for this test suite.

System	% A grade	% B grade	% C grade	% F grade
Rubi	98.89	0.23	0.41	0.48
Mathematica	74.67	6.18	17.49	1.65
Maple	52.8	22.93	7.72	16.54
Maxima	33.24	8.83	0.96	56.97
Fricas	48.52	18.03	1.51	31.93
Sympy	25.08	4.7	2.63	67.59
Giac	39.	12.5	1.01	47.49

Table 1.3:Antiderivative Grade distribution for each CAS

The following is a Bar chart illustration of the data in the above table.

The figure below compares the CAS systems for each grade level.

1.2.1 Time and leaf size Performance

The table below summarizes the performance of each CAS system in terms of time used and leaf size of results.

System	Mean time (sec)	Mean size	Normalized mean	Median size	Normalized median
Rubi	0.28	156.75	1.	107.	1.
Mathematica	1.77	800.29	2.8	92.	0.94
Maple	0.46	62669.	743.6	131.	1.27
Maxima	1.34	284.97	2.46	96.	1.36
Fricas	2.81	935.28	6.76	302.	3.43
Sympy	9.65	230.82	2.53	70.	1.14
Giac	1.53	301.08	2.55	120.	1.49

Table 1.4:Time and leaf size performance for each CAS

1.3 Performance per integrand type

The following are the different integrand types the test suite contains.

1. Algebraic Binomial problems (products involving powers of binomials and monomials).
2. Algebraic Trinomial problems (products involving powers of trinomials, binomials and monomials).
3. Miscellaneous Algebraic functions.
4. Exponentials.
5. Logarithms.
6. Trigonometric.
7. Inverse Trigonometric.
8. Hyperbolic functions.
9. Inverse Hyperbolic functions.
10. Special functions.
11. Independent tests.

The following table gives percentage solved of each CAS per integrand type.

Integrand type	problems	Rubi	Mathematica	Maple	Maxima	Fricas	Sympy	Giac
Independent tests	1892	98.31	98.73	92.18	79.39	94.34	71.78	82.72
Algebraic Binomial	14276	99.99	99.7	82.18	42.02	70.91	59.27	62.61
Algebraic Trinomial	10187	99.99	98.89	90.67	38.56	75.76	40.39	61.23
Algebraic Miscellaneous	1519	98.62	98.16	87.23	42.92	74.06	45.69	54.18
Exponentials	965	99.17	96.68	80.21	60.93	87.67	40.83	46.74
Logarithms	3085	98.51	97.8	54.49	48.36	57.76	25.32	43.37
Trigonometric	22551	99.56	97.61	85.75	41.34	63.42	13.64	44.13
Inverse Trigonometric	4585	99.65	97.97	83.84	31.15	48.29	28.16	48.05
Hyperbolic	5166	98.32	98.03	82.58	57.08	84.84	20.75	62.45
Inverse Hyperbolic	6626	99.52	98.46	80.47	40.34	62.27	24.89	39.5
Special functions	999	100.	95.4	69.97	35.54	48.85	39.34	34.93

Table 1.5:Percentage solved per integrand type

In addition to the above table, for each type of integrand listed above, 3D chart is made which shows how each CAS performed on that specific integrand type.

These charts and the table above can be used to show where each CAS relative strength or weakness in the area of integration.

1.4 Maximum leaf size ratio for each CAS against the optimal result

The following table gives the largest ratio found in each test file, between each CAS antiderivative and the optimal antiderivative.

For each test input file, the problem with the largest ratio \(\frac{\text{CAS leaf size}}{\text{Optimal leaf size}}\) is recorded with the corresponding problem number.

In each column in the table below, the first number is the maximum leaf size ratio, and the number that follows inside the parentheses is the problem number in that specific file where this maximum ratio was found. This ratio is determined only when CAS solved the the problem and also when an optimal antiderivative is known.

If it happens that a CAS was not able to solve all the integrals in the input test file, or if it was not possible to obtain leaf size for the CAS result for all the problems in the file, then a zero is used for the ratio and -1 is used for the problem number.

This makes it easy to locate the problem. In the future, a direct link will be added as well.

Table 1.6Maximum leaf size ratio for each CAS against the optimal result
file #	Rubi	Mathematica	Maple	Maxima	FriCAS	Sympy	Giac
1	1. (1)	3.9 (50)	16.6 (114)	5.1 (169)	13.7 (61)	191.1 (145)	15.5 (55)
2	7.3 (21)	6.2 (14)	3.6 (17)	2.6 (4)	55. (13)	18.5 (5)	6.2 (2)
3	1. (1)	16.1 (6)	17. (6)	14.9 (7)	11.5 (9)	1.9 (5)	2.6 (5)
4	6.4 (5)	14.3 (13)	40.7 (46)	22.4 (43)	18.7 (43)	4.4 (40)	53.7 (41)
5	1. (65)	54.7 (278)	12737.8 (278)	11. (280)	29.5 (103)	62. (12)	26.4 (141)
6	1. (1)	1. (2)	2.2 (4)	2.6 (1)	3.9 (7)	0.8 (4)	3.2 (5)
7	2.2 (3)	5.6 (7)	1.8 (3)	3.8 (3)	15.1 (9)	1.9 (7)	2.6 (3)
8	1.6 (50)	5.3 (31)	7.9 (70)	9. (11)	13. (42)	26.4 (71)	7. (70)
9	1.2 (365)	7.2 (80)	4.3 (341)	16.3 (328)	16.4 (351)	191.1 (251)	48.1 (368)
10	3.2 (335)	242.6 (327)	3343.5 (327)	49.9 (399)	109.1 (595)	76.3 (215)	25.3 (537)
11	529. (82)	127. (82)	317. (82)	3.7 (2)	150. (82)	41.3 (17)	4.9 (24)
12	1.8 (6)	2.3 (4)	1.2 (8)	2. (2)	7.8 (3)	3.4 (3)	2.1 (2)
13	7.1 (369)	23.8 (1323)	30.9 (1323)	44.4 (1323)	66.4 (1323)	136.1 (671)	45.9 (1323)
14	2. (870)	16.5 (1101)	22.6 (1101)	30. (1716)	45.7 (1101)	84.5 (67)	63. (827)
15	3.3 (97)	13.9 (72)	28.5 (100)	2.7 (155)	22.2 (21)	49.2 (119)	13.5 (119)
16	1. (1)	1.5 (17)	11. (25)	2. (10)	0. (-1)	59.5 (27)	26.8 (25)
17	2.6 (35)	10.1 (67)	39.8 (66)	2.7 (35)	12.8 (6)	5.3 (35)	17.1 (46)
18	1. (3)	27.5 (31)	68. (35)	0. (-1)	0. (-1)	0. (-1)	0. (-1)
19	8.2 (664)	6.9 (663)	7.9 (196)	13.5 (196)	22.3 (196)	55.3 (528)	14.8 (416)
20	1.6 (254)	6.4 (94)	147.4 (69)	6. (73)	62.8 (160)	21.9 (33)	14.4 (42)
21	1. (596)	12.6 (337)	46.7 (754)	4.2 (313)	49.1 (1016)	32.8 (324)	11.6 (553)
22	1.3 (64)	2.6 (63)	15.2 (57)	1.5 (18)	17.5 (60)	3. (21)	4. (98)
23	1. (1)	1.1 (50)	10.4 (15)	0. (-1)	15.6 (15)	43. (16)	18.6 (15)
24	1.2 (173)	1.9 (45)	2. (162)	3.4 (163)	12.9 (26)	18.3 (93)	5.3 (157)
25	8.4 (2686)	13.4 (2913)	141.8 (2913)	17.8 (2285)	52. (2913)	170.1 (2672)	30.8 (2813)
26	4.3 (116)	10.3 (306)	17.9 (265)	3. (47)	30.7 (265)	27.8 (238)	8.1 (292)
27	4.2 (760)	12.3 (1051)	77.4 (546)	39.3 (1063)	40.1 (317)	36.6 (124)	14.9 (1051)
28	1.2 (46)	0.9 (45)	51.1 (15)	0. (-1)	59.6 (15)	23.3 (6)	66.9 (16)
29	1.2 (552)	3.8 (45)	10.4 (43)	4.3 (161)	100.6 (416)	16.2 (171)	11. (591)
30	1.3 (278)	10. (328)	51.5 (297)	11.3 (328)	26. (348)	10. (328)	12.5 (348)
31	1. (1)	6.4 (283)	4.9 (269)	3.1 (279)	9.5 (269)	21.6 (269)	8.5 (269)
32	2.8 (83)	3.9 (25)	5.8 (74)	3.5 (98)	15.9 (127)	16.4 (63)	4.1 (74)
33	2. (2419)	23.9 (2302)	70.8 (2351)	19.2 (1497)	74.3 (2293)	72.8 (1423)	49.7 (982)
34	1.3 (1471)	15.6 (1635)	82.2 (1180)	29. (2015)	93.4 (1452)	91.8 (2147)	32.7 (2640)
35	2.1 (833)	58.6 (507)	116.1 (801)	8.1 (579)	48.7 (533)	68.8 (920)	27. (925)
36	1. (1)	10.3 (6)	425.1 (78)	3.7 (95)	61.2 (112)	1.2 (19)	9.4 (3)
37	1. (129)	9.7 (37)	14197.2 (12)	8.9 (27)	61.2 (117)	6.9 (13)	111.9 (24)
38	1.8 (76)	42.8 (204)	421. (278)	120.1 (278)	262.7 (278)	114.1 (278)	160.9 (278)
39	1.7 (636)	8.8 (109)	9.5 (885)	7.3 (515)	60.1 (1073)	28.5 (1105)	35.7 (857)
40	1.7 (212)	13.9 (409)	50.7 (220)	9. (88)	70.4 (109)	19.6 (218)	77. (32)
41	1.9 (327)	32.6 (381)	26. (136)	7.5 (70)	109.9 (305)	47.5 (220)	37.4 (109)
42	1. (59)	1.5 (25)	15.8 (54)	1.8 (111)	8.1 (46)	58.1 (87)	43.2 (21)
43	1.6 (135)	2.4 (136)	13.8 (37)	2.1 (131)	97. (60)	27.3 (39)	43. (25)
44	1.9 (1)	6.3 (24)	6.4 (29)	0. (-1)	8.4 (35)	0.8 (1)	3.3 (42)
45	1. (1)	4.9 (4)	0.9 (4)	0. (-1)	0. (-1)	0. (-1)	0. (-1)
46	2.1 (154)	12.7 (601)	54.7 (609)	8.5 (609)	99. (637)	26.3 (438)	62.9 (597)
47	1. (1)	22.3 (89)	2.7 (37)	1.8 (26)	24.2 (37)	42.2 (68)	13.2 (68)
48	1. (67)	16.1 (143)	2909.3 (93)	93. (94)	217.1 (93)	82.9 (93)	122. (93)
49	1. (1)	4.9 (17)	1.7 (11)	2.8 (16)	6. (16)	0. (-1)	4.5 (16)
50	1. (1)	1.7 (99)	4. (72)	1.2 (71)	22.1 (102)	20.7 (24)	35.7 (83)
51	6.2 (424)	11.6 (162)	1223.1 (192)	57.1 (63)	208.8 (192)	84.3 (192)	125.8 (192)
52	4.1 (997)	172.1 (1010)	3059.3 (1010)	6.8 (612)	85.5 (871)	57.5 (180)	64.3 (525)
53	1. (1)	1.2 (82)	9.5 (87)	3. (2)	4.4 (82)	2.5 (2)	75.3 (2)
54	1. (1)	1. (1)	16.8 (46)	2.2 (49)	9.5 (58)	2.2 (32)	50.6 (25)
55	1.2 (655)	5.3 (636)	38.7 (267)	176.7 (267)	35.7 (339)	11.2 (281)	73.8 (563)
56	1. (1)	1.3 (133)	83.5 (150)	2.6 (62)	11.6 (61)	4.4 (125)	13.8 (149)
57	1.7 (115)	3.9 (363)	97.5 (440)	6.9 (348)	47.1 (440)	21.2 (381)	14.6 (392)
58	1.5 (176)	12.4 (64)	504.9 (192)	3.9 (166)	20.8 (237)	3.6 (165)	18.1 (237)
59	7.5 (71)	39.2 (308)	376.9 (168)	10. (10)	14.9 (171)	5.6 (119)	15.2 (302)
60	26.4 (88)	16.8 (81)	1428.6 (228)	107.3 (81)	20.1 (228)	9.7 (28)	9. (233)
61	1.6 (79)	55.1 (51)	14.7 (74)	19.2 (44)	9.1 (15)	2.2 (50)	21.7 (34)
62	1.8 (383)	9.3 (340)	161.9 (62)	12.3 (340)	18.6 (404)	19.8 (425)	48.4 (456)
63	1.5 (390)	4.3 (45)	54.3 (175)	10. (390)	68.4 (197)	13.6 (321)	8.2 (329)
64	1.2 (284)	13.1 (44)	2190.9 (91)	14.3 (23)	23.9 (23)	15.9 (189)	20.7 (28)
65	1. (1)	114.1 (497)	33.3 (493)	5.3 (111)	13.2 (289)	32.4 (355)	7.6 (105)
66	1. (1)	8.6 (249)	7.6 (83)	25.1 (185)	29.6 (209)	30.8 (193)	74.3 (7)
67	1. (1)	9.2 (12)	4.3 (51)	3.2 (21)	9.9 (5)	17.1 (49)	5.1 (6)
68	1. (1)	7.1 (38)	7.7 (65)	28.8 (45)	5.6 (6)	2.2 (12)	74.3 (36)
69	1. (1)	3.3 (203)	7.8 (201)	227.2 (37)	10.6 (44)	8.7 (116)	7.2 (155)
70	2. (615)	79.6 (352)	447.5 (605)	12.2 (151)	55.8 (476)	56.4 (89)	30.6 (146)
71	1. (1)	1.1 (10)	1.4 (29)	20.7 (33)	3.7 (13)	3.9 (12)	3.2 (30)
72	1.6 (103)	533.6 (138)	3.6 (200)	5.4 (53)	15.8 (201)	2.6 (40)	173.8 (16)
73	1.9 (621)	1029.2 (406)	4910.9 (790)	41.5 (256)	36.3 (595)	44.2 (453)	24.3 (336)
74	1.6 (1108)	1478. (937)	173.2 (174)	11.6 (46)	35.5 (937)	69.4 (567)	107.9 (177)
75	1.3 (12)	3375. (37)	688.6 (48)	9.7 (16)	68.5 (35)	3.4 (1)	8.3 (13)
76	1.2 (206)	65.6 (202)	8067.4 (353)	47.4 (48)	38. (327)	40.6 (273)	18.1 (127)
77	1. (1)	6.7 (10)	3.9 (2)	16.8 (1)	5.6 (2)	412.4 (8)	7.3 (12)
78	1.4 (32)	72.6 (30)	4.4 (33)	4.4 (20)	5.5 (18)	2.3 (32)	1.3 (32)
79	1.8 (236)	228.2 (240)	51412.7 (593)	23.8 (487)	64. (418)	108.5 (89)	20.7 (29)
80	1. (1)	2.2 (2)	2.1 (4)	1.7 (2)	10.5 (7)	11.7 (4)	3.1 (2)
81	1. (1)	1.5 (16)	6. (13)	1.4 (19)	51.8 (1)	2.8 (11)	2.5 (14)
82	1. (1)	3.7 (284)	8.3 (12)	22.3 (170)	11.4 (176)	2.7 (64)	16.9 (64)
83	1. (1)	4. (62)	8.3 (76)	16.3 (133)	17.6 (33)	4.1 (9)	89.1 (8)
84	1. (1)	2.4 (61)	3.4 (50)	2.7 (5)	7.5 (5)	6. (41)	1.4 (19)
85	1. (1)	1.3 (94)	4.2 (26)	6.8 (40)	5.5 (36)	6. (61)	4.2 (92)
86	4.3 (11)	4.1 (60)	13.2 (78)	4.4 (3)	13.7 (11)	9.5 (1)	5. (11)
87	1. (1)	1. (10)	1.4 (29)	20.7 (32)	3.7 (13)	3.8 (12)	3.1 (30)
88	1. (1)	3.2 (1)	3.4 (3)	5.5 (3)	11.6 (20)	0. (-1)	4.1 (3)
89	1.4 (370)	35.3 (773)	9.3 (642)	50.3 (213)	15.8 (484)	12.8 (781)	66.1 (782)
90	1. (1)	2.8 (2)	2.9 (2)	0. (-1)	0. (-1)	0. (-1)	0. (-1)
91	1. (1)	3. (1)	1.8 (1)	4.7 (1)	4.2 (1)	0. (-1)	2.2 (1)
92	1.1 (40)	36.7 (454)	14.3 (436)	76.1 (95)	14.2 (270)	11.7 (38)	5.4 (80)
93	1. (1)	53.4 (393)	8. (29)	27.4 (115)	9.5 (319)	3. (9)	2.3 (5)
94	1.4 (940)	84.8 (1350)	18. (1154)	63.8 (96)	15.8 (590)	15.5 (339)	7.2 (381)
95	1.2 (81)	4.9 (91)	6.9 (70)	12.7 (53)	18. (67)	14.3 (1)	5. (91)
96	1. (1)	2.1 (9)	7.6 (21)	1.4 (2)	18. (13)	1.2 (3)	5.7 (13)
97	1. (1)	1.9 (5)	10.5 (13)	1.1 (12)	51.3 (13)	3.3 (12)	2.5 (5)
98	1. (1)	173.1 (357)	564.9 (52)	1.8 (7)	12.9 (293)	3.1 (376)	33.5 (8)
99	1. (1)	4.4 (44)	6. (54)	15. (49)	11.5 (54)	2.5 (24)	8.2 (8)
100	1. (1)	2.6 (33)	1.5 (21)	10.6 (52)	11.5 (39)	6.3 (15)	2.3 (21)
101	1.5 (562)	75.5 (641)	173.9 (617)	25.5 (393)	27.4 (80)	10.4 (88)	76.2 (540)
102	1. (1)	7. (46)	4.1 (61)	3.9 (67)	16.4 (75)	1.3 (2)	104.7 (13)
103	1.4 (891)	200.5 (678)	9426.9 (611)	21. (208)	221.2 (523)	13.3 (464)	37. (1203)
104	1. (1)	941.7 (463)	15275. (454)	21.6 (548)	188.9 (369)	20.1 (272)	32.8 (257)
105	1. (130)	3975.5 (145)	172.7 (123)	4. (83)	23.7 (83)	21.1 (70)	35.4 (51)
106	1. (1)	44.6 (159)	2905.5 (351)	24.4 (272)	61.7 (288)	30.9 (239)	104.5 (36)
107	1. (1)	777.6 (45)	36518.9 (5)	0. (-1)	34.9 (45)	0. (-1)	0. (-1)
108	1. (1)	21.6 (47)	288.2 (43)	1.7 (4)	12.5 (20)	2.6 (1)	5.7 (3)
109	1. (1)	5.5 (42)	10.1 (27)	25.1 (47)	11.2 (59)	2.4 (22)	54.5 (8)
110	1. (1)	2.5 (11)	3.4 (16)	4.4 (11)	10.3 (14)	1.3 (2)	3.5 (7)
111	1. (1)	2.4 (5)	4.3 (9)	5.8 (7)	9. (19)	1.2 (2)	3.7 (6)
112	1. (1)	3.9 (15)	69.4 (103)	2.6 (94)	11. (27)	8.9 (92)	3.2 (94)
113	1. (1)	23.7 (22)	35. (29)	18.4 (8)	32.6 (9)	21.5 (6)	7.6 (25)
114	1. (1)	1997.4 (22)	31765.8 (3)	0. (-1)	55.5 (27)	0. (-1)	0. (-1)
115	1. (1)	14.7 (42)	9.8 (259)	35. (47)	20.3 (42)	3.3 (1)	15.7 (42)
116	1. (1)	10. (40)	4.1 (29)	20. (16)	11.9 (29)	0. (-1)	8.7 (18)
117	1. (1)	3.2 (18)	5.9 (73)	97.4 (27)	10.3 (68)	2.2 (53)	2.7 (12)
118	1.4 (423)	249. (874)	14.7 (578)	70.7 (255)	15.4 (515)	2.6 (5)	5.2 (112)
119	1. (1)	45.2 (153)	12.3 (284)	3.9 (65)	12.7 (227)	0. (-1)	9.4 (196)
120	1.7 (340)	55.9 (191)	46.5 (339)	5. (67)	65.4 (338)	13.1 (90)	9.1 (286)
121	1.3 (115)	2597.8 (169)	1151.1 (153)	42.7 (108)	20.4 (159)	0. (-1)	7.2 (197)
122	2.2 (197)	1873.2 (240)	7.1 (238)	58.3 (130)	20.5 (241)	3.1 (170)	5.9 (256)
123	1.3 (265)	350.5 (634)	15.8 (385)	71. (259)	16.2 (337)	2.2 (47)	7.3 (124)
124	1. (1)	3.6 (65)	24.1 (25)	18.2 (25)	7.2 (58)	2.9 (33)	3.6 (41)
125	1.2 (870)	383.4 (1373)	19.8 (970)	62. (1289)	15.9 (808)	3.1 (930)	9.8 (490)
126	1.3 (231)	66.8 (138)	544.5 (433)	37.4 (297)	55.8 (461)	7.4 (459)	15.3 (365)
127	1. (1)	5.6 (42)	12.1 (21)	45.1 (39)	15.4 (42)	3.1 (1)	4.2 (41)
128	1. (1)	4. (25)	5.2 (74)	53.1 (15)	10.5 (69)	2.2 (53)	4. (61)
129	1. (1)	5.3 (36)	19.6 (18)	8.7 (13)	20.4 (19)	0. (-1)	18.4 (15)
130	1. (1)	2.5 (8)	4. (9)	6.6 (8)	8.9 (14)	0. (-1)	3. (8)
131	1.3 (20)	3.3 (10)	2.3 (22)	4.8 (1)	12. (22)	0. (-1)	3. (10)
132	1. (1)	2.7 (3)	2.2 (8)	3.4 (8)	5.9 (9)	4.9 (18)	4.5 (12)
133	1. (1)	1.2 (1)	1.8 (1)	0. (-1)	0. (-1)	0. (-1)	0. (-1)
134	1. (12)	3.1 (18)	26.8 (15)	32.8 (18)	40. (11)	0. (-1)	4.2 (1)
135	1. (1)	29.1 (187)	4879055.9 (170)	114.8 (57)	19.5 (231)	16.7 (241)	115.4 (200)
136	3.3 (23)	9.2 (211)	5.9 (146)	12.8 (209)	18.4 (143)	6.6 (114)	88.7 (238)
137	1.1 (281)	9.2 (164)	14.6 (80)	78.9 (391)	36.1 (273)	10.3 (396)	109.8 (293)
138	1. (1)	2.7 (1)	6.9 (9)	0.6 (5)	28.8 (4)	1.1 (5)	0.9 (5)
139	4.3 (259)	9.3 (299)	13.1 (259)	122.7 (225)	9.2 (173)	6.6 (12)	99.6 (126)
140	19.2 (34)	9.1 (133)	40.1 (34)	109.8 (34)	14.8 (63)	7.2 (135)	360.1 (31)
141	10.8 (759)	718.9 (434)	651.2 (860)	47. (927)	56.2 (503)	227.1 (140)	113.7 (904)
142	1.4 (107)	2.5 (95)	4.8 (156)	2.3 (155)	4.4 (7)	2.3 (11)	13.3 (145)
143	1.7 (100)	3.5 (200)	19.9 (90)	5.6 (195)	12.4 (642)	2.2 (9)	76.5 (620)
144	1.9 (147)	7. (85)	13.9 (55)	3.8 (186)	11.8 (93)	8.1 (206)	18.8 (233)
145	1.3 (168)	4.9 (41)	2.8 (156)	2.4 (155)	7.5 (7)	2.3 (11)	35.6 (147)
146	1. (1)	1.9 (10)	5.6 (4)	4. (11)	10.2 (33)	2. (23)	49.2 (23)
147	1. (1)	3.8 (13)	3.3 (18)	2.3 (55)	13.6 (29)	2. (58)	5.2 (31)
148	10. (146)	4.7 (83)	28.1 (148)	2.6 (165)	8. (112)	22.4 (99)	2.4 (66)
149	1.2 (25)	4.2 (25)	43.8 (20)	3.1 (21)	39.8 (21)	12.5 (21)	5.1 (8)
150	1.3 (152)	6.4 (429)	85.8 (146)	7. (218)	20.5 (1223)	24.6 (54)	4.8 (1195)
151	1. (1)	3.3 (36)	80. (56)	6.9 (1)	6.2 (30)	46.1 (5)	11.2 (14)
152	2. (344)	2.7 (248)	13.6 (329)	10.4 (375)	30. (375)	64.6 (177)	11.3 (375)
153	1.1 (117)	11.4 (54)	27.1 (147)	7.3 (67)	15. (96)	4.7 (38)	3.5 (10)
154	1.3 (109)	11.4 (164)	72.1 (110)	7.2 (164)	15. (186)	8.7 (106)	6.8 (135)
155	1. (1)	1.2 (7)	1. (2)	1.3 (2)	2.9 (5)	2.7 (4)	1.6 (2)
156	1.2 (68)	2.6 (104)	11.9 (105)	4.6 (31)	18.4 (151)	1.2 (9)	1.9 (7)
157	1. (1)	3.3 (42)	4.2 (26)	2.3 (14)	9.5 (24)	0.9 (10)	4.4 (26)
158	1.4 (51)	2.8 (111)	11.9 (112)	2.6 (22)	18.5 (156)	1.2 (9)	2. (7)
159	1. (1)	3.3 (40)	4.9 (26)	2.2 (13)	9.6 (23)	0.9 (9)	6.3 (26)
160	1. (1)	23.2 (333)	7.5 (379)	4.8 (1)	39.4 (329)	6.9 (226)	10. (259)
161	1. (1)	5.4 (53)	3.4 (98)	18.8 (90)	17.6 (20)	1.9 (10)	6. (93)
162	1. (1)	1.5 (24)	1.9 (28)	8.1 (7)	12.7 (24)	0. (-1)	2.2 (29)
163	1. (1)	30.9 (200)	8.4 (198)	18. (208)	76.9 (157)	16.8 (253)	34.9 (273)
164	1.3 (16)	9.9 (394)	15.3 (316)	29.6 (315)	175.6 (214)	96.5 (35)	21.4 (70)
165	1. (1)	13.6 (173)	7.4 (28)	4.8 (1)	39.6 (36)	4.1 (8)	4.8 (1)
166	1. (1)	7.9 (106)	3.6 (79)	4.7 (5)	9.2 (108)	2.3 (12)	4.6 (38)
167	1. (1)	2.1 (3)	3.4 (64)	18.9 (56)	17.6 (20)	1.9 (10)	6. (59)
168	1. (1)	1.5 (12)	1.9 (28)	8.1 (7)	12.6 (24)	0. (-1)	2.2 (29)
169	1. (1)	8.7 (328)	7.4 (165)	15.4 (196)	94.6 (177)	22. (152)	34.9 (246)
170	1.3 (60)	2.5 (11)	8. (38)	7.9 (40)	148.8 (12)	8.4 (37)	7. (38)
171	1. (1)	3.7 (3)	12.5 (43)	4.3 (8)	51.8 (11)	1.7 (8)	4.5 (8)
172	1.2 (109)	3.5 (212)	6.6 (102)	17. (188)	181.6 (200)	42.3 (62)	5.7 (102)
173	1.3 (257)	10.5 (252)	14.4 (114)	24.4 (104)	236.2 (244)	25.2 (183)	183.5 (109)
174	1. (1)	5.1 (48)	9.6 (22)	5. (8)	42.5 (47)	11.7 (27)	4.3 (8)
175	1. (1)	14.1 (209)	6.2 (35)	17.5 (193)	181.6 (205)	11.6 (148)	9.6 (113)
176	1. (1)	6.7 (10)	9.2 (24)	8.7 (10)	236.2 (41)	6.4 (5)	11.7 (10)
177	1. (1)	3.8 (6)	7.4 (3)	3.3 (7)	39.6 (9)	0. (-1)	3.6 (7)
178	1. (1)	3.5 (18)	3.6 (79)	2.7 (15)	60.8 (82)	0. (-1)	2.7 (31)
179	3.5 (186)	6.4 (145)	12.7 (186)	11.6 (59)	220. (136)	2.2 (119)	4.4 (86)
180	1.4 (54)	14.4 (168)	14. (169)	17.8 (124)	236.3 (209)	5.4 (142)	6.8 (167)
181	1. (1)	9.1 (26)	5. (29)	4.2 (7)	35.1 (9)	0. (-1)	3.7 (7)
182	1. (1)	5.2 (18)	3.9 (78)	3.1 (15)	75.2 (15)	0. (-1)	2.5 (5)
183	3.3 (160)	6.7 (24)	22.3 (24)	12. (91)	81.4 (124)	0. (-1)	12. (24)
184	1.1 (12)	3.4 (24)	16.5 (8)	8.4 (1)	139.3 (10)	0. (-1)	4. (1)
185	1.9 (192)	515.8 (777)	142.2 (767)	35.1 (100)	140.3 (589)	43.6 (11)	13.7 (1050)
186	1. (1)	1.9 (141)	2.7 (38)	2.3 (15)	7.9 (7)	1. (22)	3.5 (19)
187	2.1 (73)	3.5 (230)	10. (313)	9.2 (255)	12.5 (531)	3.5 (255)	4.2 (118)
188	1.2 (170)	4.7 (46)	6.2 (151)	15.6 (276)	10.9 (368)	8.2 (147)	7.7 (116)
189	1.1 (163)	3.2 (39)	2.3 (18)	1.8 (135)	4.9 (7)	1.1 (135)	3.1 (19)
190	1.7 (322)	5.5 (516)	17.2 (93)	3.7 (22)	14.3 (508)	1.7 (528)	3.5 (22)
191	1.3 (73)	6.9 (167)	26. (291)	3.4 (102)	19.3 (20)	7.6 (122)	8.2 (93)
192	8.1 (149)	1.9 (31)	7031. (175)	7.1 (202)	25.2 (216)	15.2 (61)	2.6 (60)
193	1.6 (21)	5. (12)	56.2 (20)	3.1 (40)	59.9 (32)	10.5 (24)	8.6 (8)
194	1.6 (538)	5. (156)	74.3 (235)	21.7 (244)	14.6 (516)	5.1 (61)	6.3 (505)
195	1. (43)	11.2 (42)	62.1 (46)	7. (15)	10.6 (37)	43.6 (13)	12.3 (37)
196	2. (172)	3.7 (868)	16.4 (867)	37.8 (1152)	31.8 (1368)	30.7 (997)	13. (1368)
197	1.7 (81)	24. (319)	24.6 (312)	5.8 (72)	18.9 (315)	3. (276)	6.4 (133)
198	1.2 (78)	24. (238)	18086.6 (185)	5.2 (95)	19.5 (189)	5.3 (92)	1. (298)
199	1.9 (172)	4.9 (430)	11.6 (22)	4.8 (37)	11.7 (130)	12.1 (767)	5.9 (29)
200	1. (1)	16.3 (85)	19.3 (124)	1.9 (47)	18.7 (168)	1.5 (35)	0. (-1)
201	2.8 (38)	5.6 (18)	40.5 (80)	1.3 (34)	20.9 (6)	0.7 (93)	4.1 (47)
202	1.2 (75)	2.9 (111)	11.5 (112)	2.8 (10)	20.2 (156)	1.2 (9)	0. (-1)
203	1.6 (55)	8.2 (13)	5.2 (65)	3.6 (31)	21.1 (71)	2. (31)	3.4 (31)
204	1. (1)	1.7 (102)	2.5 (221)	1.5 (31)	4.9 (140)	2.9 (83)	2.2 (18)
205	1. (1)	2.5 (57)	1.5 (92)	0. (-1)	0. (-1)	2. (179)	0. (-1)
206	1. (1)	2.6 (41)	3.3 (134)	0. (-1)	0. (-1)	7.3 (69)	84.2 (135)
207	1. (1)	2.5 (131)	1.3 (35)	0. (-1)	0. (-1)	8.5 (69)	0. (-1)
208	1.1 (174)	1.3 (195)	2.4 (144)	5.5 (155)	23.7 (150)	4.4 (29)	0. (-1)

1.5 Pass/Fail per test file for each CAS system

The following table gives the number of passed integrals and number of failed integrals per test number. There are 208 tests. Each tests corresponds to one input file.

Table 1.7Pass/Fail per test file for each CAS
Test #	Rubi		MMA		Maple		Maxima		FriCAS		Sympy		Giac
	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail	Pass	Fail
1	175	0	175	0	173	2	164	11	172	3	156	19	165	10
2	33	2	35	0	27	8	15	20	24	11	7	28	13	22
3	13	1	14	0	11	3	8	6	12	2	9	5	8	6
4	48	2	50	0	33	17	24	26	47	3	18	32	38	12
5	279	5	283	1	282	2	235	49	280	4	249	35	255	29
6	3	4	6	1	5	2	3	4	7	0	5	2	5	2
7	7	2	9	0	9	0	7	2	9	0	4	5	9	0
8	113	0	112	1	113	0	107	6	112	1	104	9	109	4
9	376	0	376	0	376	0	371	5	376	0	347	29	374	2
10	705	0	705	0	647	58	542	163	650	55	422	283	564	141
11	100	16	95	21	60	56	19	97	88	28	29	87	17	99
12	8	0	8	0	8	0	7	1	8	0	8	0	8	0
13	1917	0	1917	0	1557	360	963	954	1602	315	1173	744	1258	659
14	3201	0	3201	0	2863	338	1879	1322	2524	677	1524	1677	2330	871
15	158	1	155	4	128	31	21	138	47	112	38	121	39	120
16	34	0	34	0	28	6	12	22	0	34	14	20	28	6
17	78	0	78	0	78	0	19	59	43	35	15	63	35	43
18	35	0	35	0	35	0	0	35	0	35	0	35	0	35
19	1071	0	1071	0	755	316	395	676	674	397	970	101	639	432
20	349	0	349	0	246	103	24	325	136	213	106	243	111	238
21	1156	0	1156	0	1002	154	283	873	853	303	584	572	800	356
22	115	0	114	1	105	10	15	100	30	85	27	88	34	81
23	51	0	51	0	14	37	0	51	14	37	25	26	14	37
24	174	0	174	0	170	4	43	131	121	53	152	22	170	4
25	3078	0	3044	34	2585	493	1616	1462	2370	708	2625	453	1988	1090
26	385	0	383	2	197	188	65	320	213	172	147	238	131	254
27	1081	0	1081	0	720	361	215	866	662	419	365	716	490	591
28	46	0	46	0	12	34	0	46	12	34	10	36	11	35
29	594	0	594	0	577	17	206	388	338	256	454	140	418	176
30	454	0	454	0	385	69	136	318	256	198	113	341	234	220
31	298	0	296	2	275	23	107	191	228	70	120	178	208	90
32	143	0	143	0	113	30	93	50	113	30	47	96	104	39
33	2590	0	2584	6	2320	270	1115	1475	2065	525	1043	1547	1592	998
34	2646	0	2646	0	2584	62	1297	1349	2258	388	1182	1464	2076	570
35	958	0	937	21	729	229	179	779	569	389	261	697	268	690
36	123	0	123	0	121	2	67	56	110	13	43	80	87	36
37	143	0	142	1	141	2	12	131	61	82	10	133	47	96
38	400	0	394	6	388	12	217	183	324	76	145	255	317	83
39	1126	0	1126	0	1062	64	395	731	844	282	486	640	739	387
40	412	1	378	35	399	14	60	353	209	204	190	223	148	265
41	413	0	400	13	376	37	108	305	250	163	139	274	190	223
42	111	0	103	8	111	0	83	28	80	31	52	59	88	23
43	145	0	145	0	142	3	61	84	115	30	89	56	87	58
44	42	0	38	4	40	2	0	42	9	33	6	36	4	38
45	4	0	4	0	4	0	0	4	0	4	0	4	0	4
46	664	0	662	2	496	168	241	423	535	129	289	375	363	301
47	96	0	92	4	49	47	9	87	47	49	44	52	34	62
48	156	0	147	9	137	19	60	96	106	50	69	87	109	47
49	17	0	14	3	2	15	2	15	7	10	0	17	2	15
50	140	0	139	1	136	4	22	118	129	11	67	73	87	53
51	491	3	494	0	489	5	360	134	430	64	432	62	410	84
52	1007	18	997	28	836	189	292	733	695	330	262	763	413	612
53	98	0	98	0	78	20	64	34	87	11	38	60	56	42
54	93	0	84	9	75	18	62	31	93	0	48	45	49	44
55	766	8	751	23	621	153	462	312	666	108	308	466	346	428
56	193	0	193	0	98	95	96	97	123	70	71	122	101	92
57	456	0	449	7	309	147	158	298	280	176	195	261	197	259
58	249	0	243	6	81	168	61	188	90	159	37	212	58	191
59	288	26	298	16	187	127	238	76	206	108	55	259	157	157
60	249	14	249	14	98	165	179	84	156	107	49	214	49	214
61	106	2	108	0	25	83	65	43	39	69	15	93	38	70
62	543	4	542	5	309	238	219	328	218	329	130	417	217	330
63	641	0	621	20	338	303	273	368	392	249	119	522	336	305
64	314	0	314	0	236	78	203	111	278	36	110	204	185	129
65	538	0	538	0	442	96	200	338	274	264	98	440	212	326
66	348	0	348	0	264	84	192	156	322	26	99	249	156	192
67	72	0	72	0	47	25	32	40	39	33	32	40	33	39
68	113	0	113	0	113	0	53	60	113	0	26	87	47	66
69	357	0	345	12	245	112	223	134	305	52	93	264	156	201
70	653	0	638	15	562	91	257	396	358	295	85	568	270	383
71	36	0	36	0	34	2	34	2	36	0	20	16	34	2
72	206	2	203	5	178	30	142	66	178	30	4	204	140	68
73	837	0	820	17	635	202	216	621	512	325	123	714	323	514
74	1560	3	1519	44	1380	183	958	605	1216	347	203	1360	1116	447
75	51	0	51	0	50	1	16	35	30	21	4	47	13	38
76	358	0	348	10	289	69	132	226	275	83	60	298	160	198
77	19	0	15	4	12	7	13	6	13	6	8	11	12	7
78	34	0	32	2	5	29	7	27	8	26	1	33	1	33
79	590	4	583	11	521	73	174	420	393	201	61	533	290	304
80	9	0	9	0	9	0	2	7	9	0	5	4	9	0
81	19	0	19	0	19	0	5	14	15	4	6	13	9	10
82	294	0	294	0	195	99	92	202	93	201	13	281	27	267
83	189	0	187	2	134	55	134	55	133	56	50	139	80	109
84	62	0	62	0	45	17	37	25	39	23	32	30	34	28
85	99	0	99	0	87	12	77	22	91	8	30	69	47	52
86	88	0	88	0	88	0	27	61	32	56	21	67	32	56
87	34	0	34	0	32	2	32	2	34	0	18	16	32	2
88	22	0	22	0	22	0	15	7	21	1	0	22	20	2
89	932	0	923	9	854	78	285	647	443	489	77	855	254	678
90	4	0	4	0	4	0	0	4	0	4	0	4	0	4
91	1	0	1	0	1	0	1	0	1	0	0	1	1	0
92	644	0	634	10	634	10	189	455	313	331	61	583	199	445
93	393	0	389	4	236	157	119	274	121	272	7	386	15	378
94	1541	0	1534	7	1532	9	451	1090	708	833	120	1421	525	1016
95	98	0	98	0	98	0	32	66	70	28	15	83	67	31
96	21	0	21	0	21	0	2	19	15	6	2	19	19	2
97	20	0	20	0	20	0	4	16	16	4	5	15	9	11
98	387	0	386	1	265	122	70	317	136	251	13	374	68	319
99	62	1	63	0	58	5	43	20	63	0	26	37	35	28
100	66	0	66	0	36	30	43	23	48	18	32	34	36	30
101	700	0	700	0	580	120	362	338	452	248	104	596	250	450
102	91	0	90	1	83	8	74	17	83	8	5	86	76	15
103	1328	0	1208	120	1113	215	353	975	910	418	246	1082	544	784
104	855	0	797	58	780	75	310	545	583	272	174	681	331	524
105	171	0	169	2	122	49	84	87	84	87	44	127	78	93
106	499	0	497	2	408	91	189	310	403	96	81	418	283	216
107	51	0	51	0	40	11	0	51	16	35	0	51	0	51
108	52	0	52	0	37	15	22	30	21	31	8	44	14	38
109	61	0	61	0	58	3	39	22	61	0	27	34	35	26
110	23	0	23	0	23	0	13	10	18	5	5	18	22	1
111	19	0	19	0	19	0	11	8	15	4	4	15	19	0
112	106	0	105	1	103	3	3	103	31	75	1	105	3	103
113	64	0	64	0	63	1	16	48	64	0	11	53	46	18
114	32	0	32	0	25	7	0	32	16	16	0	32	0	32
115	299	0	299	0	225	74	93	206	106	193	16	283	36	263
116	46	0	45	1	42	4	24	22	46	0	20	26	24	22
117	83	0	79	4	51	32	27	56	63	20	35	48	43	40
118	879	0	865	14	735	144	309	570	393	486	30	849	289	590
119	305	1	301	5	267	39	175	131	191	115	2	304	191	115
120	364	1	341	24	331	34	162	203	241	124	35	330	228	137
121	240	1	225	16	216	25	95	146	145	96	3	238	67	174
122	286	0	273	13	262	24	166	120	232	54	1	285	192	94
123	634	0	634	0	586	48	193	441	299	335	7	627	204	430
124	70	0	70	0	70	0	48	22	49	21	3	67	46	24
125	1373	0	1343	30	1263	110	459	914	728	645	11	1362	531	842
126	468	2	424	46	431	39	140	330	401	69	19	451	242	228
127	70	0	70	0	53	17	28	42	31	39	7	63	28	42
128	84	0	80	4	50	34	29	55	64	20	35	49	44	40
129	59	0	53	6	41	18	25	34	41	18	2	57	38	21
130	16	0	16	0	16	0	12	4	16	0	0	16	16	0
131	23	0	23	0	23	0	18	5	23	0	0	23	23	0
132	24	0	24	0	24	0	24	0	24	0	9	15	24	0
133	1	0	1	0	1	0	0	1	0	1	0	1	0	1
134	27	0	27	0	27	0	11	16	27	0	0	27	12	15
135	254	0	252	2	224	30	159	95	209	45	38	216	152	102
136	294	0	294	0	289	5	230	64	290	4	61	233	282	12
137	397	0	397	0	359	38	291	106	365	32	102	295	222	175
138	9	0	9	0	9	0	1	8	9	0	1	8	1	8
139	254	76	305	25	100	230	139	191	119	211	55	275	76	254
140	140	2	142	0	114	28	58	84	115	27	32	110	63	79
141	944	6	938	12	907	43	622	328	845	105	392	558	696	254
142	227	0	226	1	217	10	63	164	79	148	91	136	168	59
143	700	3	694	9	554	149	190	513	265	438	172	531	280	423
144	472	2	459	15	378	96	85	389	182	292	155	319	241	233
145	227	0	227	0	215	12	64	163	79	148	92	135	168	59
146	33	0	33	0	30	3	10	23	15	18	11	22	13	20
147	118	0	113	5	78	40	26	92	48	70	31	87	50	68
148	157	9	163	3	144	22	87	79	91	75	84	82	98	68
149	29	2	26	5	30	1	14	17	11	20	9	22	14	17
150	1301	0	1279	22	1200	101	389	912	552	749	395	906	726	575
151	70	0	67	3	69	1	32	38	28	42	23	47	28	42
152	385	0	368	17	203	182	111	274	286	99	66	319	113	272
153	153	0	153	0	133	20	86	67	131	22	42	111	67	86
154	234	0	229	5	228	6	132	102	168	66	78	156	108	126
155	12	0	12	0	6	6	1	11	6	6	3	9	1	11
156	174	0	169	5	139	35	60	114	108	66	19	155	46	128
157	50	0	49	1	37	13	14	36	28	22	3	47	17	33
158	178	0	176	2	147	31	53	125	110	68	14	164	48	130
159	49	0	49	0	36	13	11	38	27	22	3	46	17	32
160	502	0	465	37	341	161	258	244	454	48	94	408	196	306
161	102	0	101	1	80	22	83	19	78	24	29	73	43	59
162	33	0	33	0	31	2	31	2	33	0	9	24	31	2
163	369	0	369	0	318	51	210	159	304	65	108	261	267	102
164	525	0	501	24	488	37	196	329	366	159	68	457	269	256
165	183	0	181	2	108	75	131	52	150	33	53	130	101	82
166	111	0	111	0	111	0	64	47	111	0	26	85	71	40
167	68	0	68	0	58	10	60	8	60	8	21	47	38	30
168	33	0	33	0	31	2	31	2	33	0	8	25	31	2
169	336	0	335	1	293	43	206	130	283	53	90	246	256	80
170	85	0	84	1	85	0	20	65	67	18	14	71	44	41
171	72	5	71	6	69	8	61	16	64	13	29	48	46	31
172	206	41	247	0	207	40	151	96	209	38	64	183	177	70
173	263	0	263	0	249	14	135	128	246	17	40	223	215	48
174	61	0	60	1	58	3	53	8	61	0	28	33	35	26
175	183	41	224	0	164	60	105	119	177	47	31	193	126	98
176	53	0	53	0	43	10	12	41	50	3	5	48	22	31
177	16	0	16	0	8	8	5	11	12	4	3	13	4	12
178	84	0	80	4	50	34	39	45	63	21	33	51	44	40
179	201	0	192	9	140	61	90	111	142	59	8	193	113	88
180	220	0	220	0	180	40	105	115	211	9	10	210	147	73
181	29	0	29	0	19	10	12	17	25	4	4	25	8	21
182	83	0	75	8	49	34	49	34	62	21	34	49	43	40
183	175	0	175	0	136	39	92	83	130	45	0	175	106	69
184	27	0	27	0	14	13	10	17	24	3	0	27	14	13
185	1059	0	1051	8	936	123	740	319	968	91	263	796	779	280
186	156	0	156	0	109	47	49	107	43	113	45	111	45	111
187	663	0	663	0	506	157	225	438	241	422	160	503	175	488
188	370	1	370	1	249	122	65	306	147	224	86	285	94	277
189	166	0	166	0	112	54	55	111	52	114	50	116	53	113
190	569	0	558	11	476	93	209	360	236	333	109	460	185	384
191	295	1	288	8	197	99	51	245	122	174	72	224	79	217
192	216	27	231	12	196	47	106	137	146	97	74	169	124	119
193	46	3	47	2	48	1	20	29	15	34	8	41	16	33
194	538	0	536	2	508	30	258	280	256	282	138	400	171	367
195	62	0	60	2	61	1	30	32	17	45	9	53	16	46
196	1378	0	1353	25	1100	278	515	863	1110	268	445	933	728	650
197	361	0	361	0	342	19	238	123	338	23	80	281	257	104
198	300	0	294	6	273	27	227	73	223	77	95	205	29	271
199	935	0	914	21	771	164	451	484	829	106	193	742	527	408
200	190	0	185	5	154	36	60	130	120	70	38	152	45	145
201	100	0	98	2	74	26	21	79	69	31	1	99	4	96
202	178	0	173	5	103	75	51	127	113	65	14	164	46	132
203	71	0	71	0	53	18	42	29	49	22	32	39	23	48
204	311	0	300	11	179	132	139	172	258	53	151	160	133	178
205	218	0	190	28	154	64	60	158	60	158	106	112	60	158
206	136	0	134	2	118	18	34	102	34	102	50	86	106	30
207	136	0	136	0	104	32	34	102	34	102	50	86	34	102
208	198	0	193	5	144	54	88	110	102	96	36	162	16	182

1.6 Timing

The command AboluteTiming[] was used in Mathematica to obtain the elapsed time for each integrate call. In Maple, the command Usage was used as in the following example

cpu_time := Usage(assign ('result_of _int',int(expr,x)),output='realtime'

For all other CAS systems, the elapsed time to complete each integral was found by taking the difference between the time after the call has completed from the time before the call was made. This was done using Python’s time.time() call.

All elapsed times shown are in seconds. A time limit of 3 minutes was used for each integral. If the integrate command did not complete within this time limit, the integral was aborted and considered to have failed and assigned an F grade. The time used by failed integrals due to time out is not counted in the final statistics.

1.7 Verification

A verification phase was applied on the result of integration for Rubi and Mathematica. Future version of this report will implement verification for the other CAS systems. For the integrals whose result was not run through a verification phase, it is assumed that the antiderivative produced was correct.

Verification phase has 3 minutes time out. An integral whose result was not verified could still be correct. Further investigation is needed on those integrals which failed verifications. Such integrals are marked in the summary table below and also in each integral separate section so they are easy to identify and locate.

1.8 Important notes about some of the results

1.8.1 Important note about Maxima results

Since these integrals are run in a batch mode, using an automated script, and by using sagemath (SageMath uses Maxima), then any integral where Maxima needs an interactive response from the user to answer a question during evaluation of the integral in order to complete the integration, will fail and is counted as failed.

The exception raised is ValueError. Therefore Maxima result below is lower than what could result if Maxima was run directly and each question Maxima asks was answered correctly.

The percentage of such failures were not counted for each test file, but for an example, for the Timofeev test file, there were about 30 such integrals out of total 705, or about 4 percent. This pecrentage can be higher or lower depending on the specific input test file.

Such integrals can be indentified by looking at the output of the integration in each section for Maxima. If the output was an exception ValueError then this is most likely due to this reason.

Maxima integrate was run using SageMath with the following settings set by default

'besselexpand : true'
'display2d : false'
'domain : complex'
'keepfloat : true'
'load(to_poly_solve)'
'load(simplify_sum)'
'load(abs_integrate)' 'load(diag)'

SageMath loading of Maxima abs_integrate was found to cause some problem. So the following code was added to disable this effect.

 from sage.interfaces.maxima_lib import maxima_lib
 maxima_lib.set('extra_definite_integration_methods', '[]')
 maxima_lib.set('extra_integration_methods', '[]')

See https://ask.sagemath.org/question/43088/integrate-results-that-are-different-from-using-maxima/ for reference.

1.8.2 Important note about FriCAS and Giac/XCAS results

There are Few integrals which failed due to SageMath not able to translate the result back to SageMath syntax and not because these CAS system were not able to do the integrations.

These will fail With error Exception raised: NotImplementedError

The number of such cases seems to be very small. About 1 or 2 percent of all integrals.

Hopefully the next version of SageMath will have complete translation of FriCAS and XCAS syntax and I will re-run all the tests again when this happens.

1.8.3 Important note about finding leaf size of antiderivative

For Mathematica, Rubi and Maple, the buildin system function LeafSize is used to find the leaf size of each antiderivative.

The other CAS systems (SageMath and Sympy) do not have special buildin function for this purpose at this time. Therefore the leaf size is determined as follows.

For Fricas, Giac and Maxima (all called via sagemath) the following code is used

#see https://stackoverflow.com/questions/25202346/how-to-obtain-leaf-count-expression-size-in-sage

def tree(expr):
    if expr.operator() is None:
       return expr
    else:
       return [expr.operator()]+map(tree, expr.operands())

try:
    # 1.35 is a fudge factor since this estimate of leaf count is bit lower than
    #what it should be compared to Mathematica's
    leafCount = round(1.35*len(flatten(tree(anti))))
except Exception as ee:
    leafCount =1

For Sympy, called directly from Python, the following code is used

try:
  # 1.7 is a fudge factor since it is low side from actual leaf count
  leafCount = round(1.7*count_ops(anti))

  except Exception as ee:
         leafCount =1

When these cas systems have a buildin function to find the leaf size of expressions, it will be used instead, and these tests run again.

1.9 Design of the test system

The following diagram gives a high level view of the current test build system.