Getting StartedIntroductionGetting startedOverviewRunning TestoTests running policy
Testo for Hyper-V GuidesGuide 1. The very first testGuide 2. Ubuntu Server InstallationGuide 3. Internet access in virtual machines
Testo for QEMU GuidesGuide 1. The very first testGuide 2. Ubuntu Server InstallationGuide 3. Guest AdditionsGuide 4. ParamsGuide 5. CachingGuide 6. Internet access in virtual machinesGuide 7. Linking up virtual machinesGuide 8. Flash DrivesGuide 9. MacrosGuide 10. If clauseGuide 11. No snapshotsGuide 12. Mouse controlGuide 13. Importing disk imagesGuide 14. JS-selectorsGuide 15. CyclesGuide 16. Macros with declarations
ReferenceTest scripts general overviewTesto lang basic syntaxVirtual Machine declarationVirtual flash drive declarationVirtual network declarationParamsTests declarationMacrosVirtual Machines ActionsMouse actionsDetecting images on the screenVirtual Flash Drives ActionsConditionsCyclesPossible keyboard key IDs
Tests running policy
In Testo Framework tests are hierarchical and base on the "testing from the simple to the complex" concept. The more complex test relies on the successful run of the simplier test and runs only when the simple test is complete.
In the figure above the tests
T3 are the most basic and simple, they are to be run in any case. The
T4 test is run only if (and when) the test
T1 completes with a success (tests
Т10 have the similar situation). The test
T8 requires the success of the tests
T9 is the most complex and requires all the other tests to be completed successfully.
You can specify the tests to be run with the
test_spec <wildcard match>command line argument. In this case Testo will run only the tests which names match the template
<wildcard match>. You could also exclude certain tests from running with the
exclude <wildcard match>argument.
All tests have one or more virtual machines (VM). A child-test may use either the exact set of VMs from its parent tests (tests
T7 on the fig. 1) or the exact set of VMs plus one or more new virtual machines that were not used before (test
The moment a VM is referenced in a test, it becomes bound to this test. From now on, this VM is bound to all the children-tests of this test, even if there's no new references of this virtual machine, ever. For example, the virtual machine
VM3 is first referenced in the test
VM3 is bound to the test
T5 also has the childen
T9, so they inherit the
VM3 binding, even if they have no commands involving
In Testo Framework the situation when a virtual machine is bound to more than one parent-test is prohibited.
In the example above
VM1 was referencd in the
T6 test. This situation is treated as an error: both parents for the test
T9 have the same virtual machine
VM1 bound to them.
Virtual flash drives share the policy of referencing restrictions with virtual machines. When a flash drive is plugged or referenced during a test, this flash drive becomes bound to the test and all its children. Similar to the virtual machines restriction, the situation when a flash drive is bound to more than one parent-test is prohibited.
Tests running could take quite a lot of time. So, to save the efforts and valuable users' time, Testo Framework implements the tests caching technology. This technology gives the ability to run only the tests that should be run for some reasons. See this page for the detailed list of these reasons.
The tests running process is similar to software's incrimental compilation: to build a programm you need to recompile only the outdated or not-precompiled sources.
By default, at the end of each successfully run test, for every test-bound virtual machine a snapshot is created. This snapshot is used in the future for "incrimental" tests runs.
Let's consider the following example. Take a look at the fig. 1. Let's assume that we need to run all the tests and we're doing this for the first time. In this case Testo will perform "the full run", from the bottom to the top, according to the hierarchy. If there are no errors, then at the end of each successful test, snapshots for every bound VM and virtual flash drive are created. The tests themselves is marked "cached". If the tests are run one more time after that, the run will complete instantly, because all the tests are cached now (there was no significant changes since the last successful run).
For example, if the test
T4 loses the cache for any reason, then all the
T4's children (
T9) lose their cache as well. In this case the second run will cause Testo to restore the snapshots for the VMs
VM4 and rollback them to the state they were at the end of the tests
T8. All the un-cached tests will be run:
You can invalidate the tests cache manually by using the
invalidate <wildcard match>command line argument. All the children-tests will lose their cache automatically.
Likewise, Testo creates snapshots for virtual flash drives referenced in a test. This way you can always be sure that all the flash drives are in the exact state you expect them to be. Any changes to the flash drives in other tests will be canceled.
Before actually running the tests, the interpreter asks interactively the user for the confirmation to run the tests which lost their cache. It is done because test runs could take a really long time, so we want to give the user a chance to reconsider the run if the cache loss is caused by some kind of miskate. You can disable this warning with the
--assume_yescommand line argument.
With the tests amount piling up, the situation becomes more and more demanding for the disk space being consumed. To save as much disk space as possible, you can create tests without hypervisor snapshots. To do so you must specify the
[no_snapshots: true] attribute just before the test declaration. In this case no hypervisor snapshots are going to be created at the end of the test for the virtual machines bound to the test. This can save you a lot of disk space.
The absence of the hypervisor snapshots does not mean the absence of caching - those are completely different and independent mechanisms in Testo Framework. And hence, a test marked with the
no_snaphots: trueattribute is still cached all the same and still not going to run second time (without a reason to do so).
Tests without the hypervisor snapshots let you save disk space, but there's a downside to this mechanism as well: no-snapshots tests cannot be used as a "starting point" in inrecemntal tests running. For example, if the test
T4 (fig. 1) is marked
no_snapshots: true, and the test
T7 loses its cache for some reason, then instead of rolling back
VM1 to the end of
T4 state, Testo will be forced to rollback
VM1 to the
T1 state (if
T1 is not marked as
no_snapshots) and re-run the test
T4, even though it has a valid cache. This behaviour is required to get the VMs in the right state before running the lost-cache test.
no_snapshotsmechanism lets you generally adjust the "tests run speed - disk space saving" ratio. Some tests may be marked as
no_snapshotswithout any drawbacks, and other tests shouldn't be marked as such because it would be too costly. You should stick to the general rule: tests writer should choose the "anchor" tests. The "anchor" test means that its results will be often restored when doing incremental testing (their cache looks solid and won't be lost too often). These tests are strictly advised to have the hypervisor snapshots. More volatile tests (cache is lost often enough) may be marked as
You can see
no_snapshotsattribute management in action here.