Skip to content

Commit 681ef77

Browse files
derickrderickr
committed
1 parent eb048ce commit 681ef77

File tree

3 files changed

+165
-4
lines changed

3 files changed

+165
-4
lines changed

‎docs/source/index.rst

+1
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121

2222
miscellaneous/stubs
2323
miscellaneous/writing-tests
24+
miscellaneous/running-tests
2425

2526
Welcome to the php-src documentation!
2627

‎docs/source/miscellaneous/running-tests.rst

+160
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
###############
2+
Running Tests
3+
###############
4+
5+
The easiest way to test your PHP build is to run make test from the command line after successfully
6+
compiling. This will run the all tests for all enabled functionalities and extensions located in
7+
tests folders under the source root directory using the PHP CLI binary.
8+
9+
``make test`` executes the ``run-tests.php`` script under the source root (parallel builds will not
10+
work). Therefore you can execute the script as follows:
11+
12+
.. code:: shell
13+
14+
TEST_PHP_EXECUTABLE=sapi/cli/php \
15+
sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]
16+
17+
******************************************
18+
Which php executable does make test use?
19+
******************************************
20+
21+
If you are running the ``run-tests.php`` script from the command line (as above) you must set the
22+
``TEST_PHP_EXECUTABLE`` environment variable to explicitly select the PHP executable that is to be
23+
tested, that is, used to run the test scripts.
24+
25+
If you run the tests using make test, the PHP CLI and CGI executables are automatically set for you.
26+
``make test`` executes ``run-tests.php`` script with the CLI binary. Some test scripts such as
27+
session must be executed by CGI SAPI. Therefore, you must build PHP with CGI SAPI to perform all
28+
tests.
29+
30+
**Note:** The PHP binary executing ``run-tests.php`` and the PHP binary used for executing test
31+
scripts may differ. If you use different PHP binary for executing ``run-tests.php`` script, you may
32+
get errors.
33+
34+
************************
35+
Which php.ini is used?
36+
************************
37+
38+
``make test`` uses the same ``php.ini`` file as it would once installed. The tests have been written
39+
to be independent of that ``php.ini`` file, so if you find a test that is affected by a setting,
40+
please report this, so we can address the issue.
41+
42+
**********************************
43+
Which test scripts are executed?
44+
**********************************
45+
46+
The ``run-tests.php`` (``make test``), without any arguments executes all test scripts by extracting
47+
all directories named tests from the source root and any subdirectories below. If there are files,
48+
which have a phpt extension, ``run-tests.php`` looks at the sections in these files, determines
49+
whether it should run it, by evaluating the ``SKIPIF`` section. If the test is eligible for
50+
execution, the ``FILE`` section is extracted into a ``.php`` file (with the same name besides the
51+
extension) and gets executed. When an argument is given or ``TESTS`` environment variable is set,
52+
the GLOB is expanded by the shell and any file with extension ``*.phpt`` is regarded as a test file.
53+
54+
Tester can easily execute tests selectively with as follows:
55+
56+
.. code:: shell
57+
58+
./sapi/cli/php run-tests.php ext/mbstring/*
59+
./sapi/cli/php run-tests.php ext/mbstring/020.phpt
60+
61+
**************
62+
Test results
63+
**************
64+
65+
Test results are printed to standard output. If there is a failed test, the ``run-tests.php`` script
66+
saves the result, the expected result and the code executed to the test script directory. For
67+
example, if ``ext/myext/tests/myext.phpt`` fails to pass, the following files are created:
68+
69+
- ``ext/myext/tests/myext.php`` - actual test file executed
70+
- ``ext/myext/tests/myext.log`` - log of test execution (L)
71+
- ``ext/myext/tests/myext.exp`` - expected output (E)
72+
- ``ext/myext/tests/myext.out`` - output from test script (O)
73+
- ``ext/myext/tests/myext.diff`` - diff of .out and .exp (D)
74+
75+
Failed tests are always bugs. Either the test is bugged or not considering factors applying to the
76+
tester's environment, or there is a bug in PHP. If this is a known bug, we strive to provide bug
77+
numbers, in either the test name or the file name. You can check the status of such a bug, by going
78+
to: ``https://bugs.php.net/12345`` where 12345 is the bug number. For clarity and automated
79+
processing, bug numbers are prefixed by a hash sign '#' in test names and/or test cases are named
80+
``bug12345.phpt``.
81+
82+
**Note:** The files generated by tests can be selected by setting the environment variable
83+
``TEST_PHP_LOG_FORMAT``. For each file you want to be generated use the character in brackets as
84+
shown above (default is LEOD). The php file will be generated always.
85+
86+
**Note**: You can set environment variable ``TEST_PHP_DETAILED`` to enable detailed test
87+
information.
88+
89+
*******************
90+
Automated testing
91+
*******************
92+
93+
If you like to keep up to speed, with latest developments and quality assurance, setting the
94+
environment variable ``NO_INTERACTION`` to 1, will not prompt the tester for any user input.
95+
96+
Normally, the exit status of make test is zero, regardless of the results of independent tests. Set
97+
the environment variable ``REPORT_EXIT_STATUS`` to ``1``, and make test will set the exit status
98+
("$?") to non-zero, when an individual test has failed.
99+
100+
Example script to be run by cron:
101+
102+
.. code:: shell
103+
104+
========== qa-test.sh =============
105+
#!/bin/sh
106+
107+
CO_DIR=$HOME/cvs/php7
108+
[email protected]
109+
TMPDIR=/var/tmp
110+
TODAY=`date +"%Y%m%d"`
111+
112+
# Make sure compilation environment is correct
113+
CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
114+
export MAKE=gmake
115+
export CC=gcc
116+
117+
# Set test environment
118+
export NO_INTERACTION=1
119+
export REPORT_EXIT_STATUS=1
120+
121+
cd $CO_DIR
122+
cvs update . >>$TMPDIR/phpqatest.$TODAY
123+
./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
124+
$MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
125+
if test $? -gt 0
126+
then
127+
cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
128+
fi
129+
========== end of qa-test.sh =============
130+
131+
**Note:** The exit status of ``run-tests.php`` will be ``1`` when ``REPORT_EXIT_STATUS`` is set. The
132+
result of make test may be higher than that. At present, gmake 3.79.1 returns 2, so it is advised to
133+
test for non-zero, rather then a specific value.
134+
135+
When ``make test`` finished running tests, and if there are any failed tests, the script asks to
136+
send the logs to the PHP QA mailing list. Please answer ``y`` to this question so that we can
137+
efficiently process the results, entering your e-mail address (which will not be transmitted in
138+
plain text to any list) enables us to ask you some more information if a test failed. Note that this
139+
script also uploads php -i output so your hostname may be transmitted.
140+
141+
Specific tests can also be executed, like running tests for a certain extension. To do this you can
142+
do like so (for example the standard library):
143+
144+
.. code:: shell
145+
146+
make test TESTS=ext/standard.
147+
148+
Where ``TESTS=`` points to a directory containing .phpt files or a single .phpt file like:
149+
150+
.. code:: shell
151+
152+
make test TESTS=tests/basic/001.phpt.
153+
154+
You can also pass options directly to the underlying script that runs the test suite
155+
(``run-tests.phpt``) using ``TESTS=``, for example to check for memory leaks using Valgrind, the
156+
``-m`` option can be passed along: ``make test TESTS="-m Zend/"``. For a full list of options that
157+
can be passed along, then run ``make test TESTS=-h``.
158+
159+
*Windows users:* On Windows the ``make`` command is called ``nmake`` instead of ``make``. This means
160+
that on Windows you will have to run ``nmake test``, to run the test suite.

‎docs/source/miscellaneous/writing-tests.rst

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
1-
############
2-
Test Files
3-
############
1+
###############
2+
Writing Tests
3+
###############
44

55
******************
66
phpt Test Basics
@@ -501,7 +501,7 @@ configurations, preventing your test from failing due inconsistencies in the err
501501
Alternatively you can use ``--EXPECTF--`` and check for the message by replacing the path of the
502502
source of the message with ``%s`` and the line number with ``%d``. The end of a message in a test
503503
file ``example.phpt`` then looks like ``in %sexample.php on line %d``. We explicitly dropped the
504-
last path devider as that is a system dependent character ``/`` or ``\``.
504+
last path divider as that is a system dependent character ``/`` or ``\``.
505505

506506
Last bit
507507
========

0 commit comments

Comments
 (0)