Knowing Example: Software Tests

Once a human has put knowledge into software, then that knowledge can exist independently of the human.  As example of how this works, I will use the activity of testing software since it is something about which I am familiar.

Knowing Example: Designing Tests to Prove a Software Program is "Correct"

(a)    What the program should do

This requires a person to understand (read: acquire knowledge of): the functions the software is supposed to perform: specifically what inputs it should take, what it should do with them, and what outputs it should create.  These are what it should do, not what it actually does do, of course—what it actually does is what the test is supposed to uncover.  
It can be helpful, though not necessary, for the test designer to have some understanding of what the software does do, how it was structured, what its limitations in terms of memory and throughput, etc., might be [1].  For this, the tester requires access to appropriate sources of knowledge: user requirements, systems design, code, etc.  There are plentiful heuristics that assist a test designer in creating high quality tests given such information.  There are only a limited number of things we expect a system to do so the set of all things any system of any size should do is very large, but it is finite and bounded.

(b)    What the program should not do

This is a lot harder.  It is also a lot bigger—as in infinite [2].  While there are a lot of things: inputs, input combinations, process states, outputs, etc., that system should do, there are an infinite number of things a system should not do (such as disturbing the orbit of the planet Jupiter).  This places a great strain on the tester.  Devising tests to show that a system does what it should is difficult enough.  Devising a test for something it shouldn’t do requires knowing all the things it shouldn’t do and so the tester must devise a test to expose something which he or she is not actually looking for.  It is hard.

(c)     How to test the program
In addition to what it should and shouldn’t do, the test designer needs to devise a process for testing.  Sometimes this is simply a schedule of what runs when and what should result.  Sometimes it requires building a test system.  Of course, the test system itself will have requirements, it must be designed, built, and deployed.  And tested.  And the tests for the test system will themselves require testing to prove they are correct...

Proving and Storing Example: Running the Tests

Once the above is done, the tests themselves need to be run.  This usually requires setting up the input, feeding it into the system-to-be-tested, monitoring the execution, and checking the output against a set of expectations.  If the system operation sufficiently conforms to these expectations the assumption (or more correctly: hope) is that the system under test does, in fact, contain the correct knowledge.

Here we see some of the knowledge issues.  There is knowledge within the software which the tester is trying to prove.  There is also knowledge in the source documents which may or may not be “correct.”   There is knowledge in the expected outputs which may or may not be correct.  There is knowledge in the test system which may or may not be correct.  There is knowledge applied in checking the results of the tests which may be perfect or may be flawed.  There is higher-order knowledge that must be applied to the tests: since all testing is sample testing—checking a tiny fraction of the universe of all possible tests—there must be some consideration of how well the test sample actually represents the universe of all possible. 

Where test systems are required, the test system itself must be tested, otherwise there is no assurance that it works correctly and doesn’t throw false positives and negatives, or simply misses required tests entirely. 

So “testing” of the knowledge content of everything used or needed in testing is itself required.  And, of course, the practical questions are always: against what?

We have already looked at this recursive issue, so there is no need to revisit here.  Let us assume that there is a process and benchmark source considered sufficiently complete to be able to assert that a test is “correct.”  Examination of this process always requires the intervention of a human.  Even if an “automated checker” is employed, the rules for that checker were, at some point, set up and approved by a human.

Early Testing: Must be Manual
Early software testing usually requires a small number of tests; there are many things that must work correctly to get any result from a system, positive or negative, so the human-cognitive bandwidth limitations are managed by building small, often simple, tests.  This allows the humans to assert the tests are “good”, useful, and representative of the universe.  They also help show that the test system is functioning sufficiently within bounds, the checking process can be managed, etc. 
Later Testing: Can be Automated
However, once the tests themselves have “passed” (whether the thing being tested has passed or not), the tests can be packaged into an executable software form, usually called a “test system.”  This allows them to be re-run with minimal effort on the part of the testers.  It also allows “good” and proven tests to be merged with other good and proven tests to create more complex and valuable test scenarios.  All while minimizing the effort on the part of the testers.

This frees the testers to move on to creating and assessing the effectiveness of other tests which, in their turn, can be packaged into executable test systems.  

Identifying Knowledge, Storing Knowledge, Executing Knowledge

Here we see the dividing line that, at present, differentiates the boundary between knowledge and the knower.  

The identification of what is knowledge is (currently) restricted to humans and always requires their intervention at some point.  Once a human has certified that it is, in fact, correct knowledge, the storing and execution of it can be done by machines without humans being involved, except perhaps to ensure the test systems have started and are running (though that could also be automated).

Human society is on the cusp of a revolution through which this may change.  From a functional perspective, machine intelligence (AI) is approaching the ability of humans to recognize and perhaps create knowledge that machines can use.  When this happens, knowledge will be created, stored, and executed without the intervention of people.  This will be a true paradigm shift where, on this side of the revolution, we cannot see what it will be like during and after. 

FOOTNOTES

[1] There is a philosophy of software testing, called Clean Room Testing, that asserts that knowing what the software can do and does do will bias testing and you get better testing results if you don't know anything about how and why the program was created.

[2] The Halting Problem (Einscheidungsproblem) identified by Alonzo Church and Alan Turing (among others) shows that even simple looping computational systems cannot be guaranteed to finish for any and all inputs and therefore are functionally infinite.