Cort Dougan
``Good programmers are lazy'' is a phrase that is repeated and adhered to as much as ``go to use considered harmful'' is. There is rarely understanding behind either statement, though. Few people who quote Dijkstra's paper[#!dijkstra!#] on the use of goto have even read it (even worse, few actually thought about it). Instead, they've heard the phrase and take the literal meaning of it as a guide instead of studying the actual cautions presented in the paper.
I've met and worked with lazy programmers and every one of them have been very bad programmers. A better statement would be ``Good programmers find efficient, maintainable and throughly thought out solutions to problems''. This admonishment to be lazy is usually made by some well meaning and ignorant instructors while students are studying in college. I was given the same advice when I was a student but I have learned better since then. Good programmers are hard working, methodical, determined and driven - not lazy. The path of least resistance is for dairy cows, not for intelligent engineers and programmers.
As a means of comparison, what would you think if you were told we put an airplane together and it worked when we taxied around the airport so we know it's ready to fly now. It's a bit of an exaggeration but it's closer to the methods employed and taught by computer programmers than those taught and used by aeronautical engineers.
What I suggest is that computer scientists should be taught that one must ``prove'' an algorithm or program works in theory, ``prove'' it works works in simulation and then ``prove'' it works as a real program.
Students must also be taught that even then ``proof'' is not real - even in the world of computer science or mathematics. Algorithms have been ``proven'' to be right and then found to be actually wrong[#!sha!#]. There are errors in mathematical models, there are errors in understanding of the problem being solved and there are errors in implementation. People are sometimes wrong, designs are sometimes wrong and nothing one can do will ever completely eliminate that possibility. We must strive to remove as much chance of error as possible in our methods. We cannot accept that any set of tests or proofs will be a guarantee. Testing can demonstrate the presence of bugs but not their absence[#!testing!#].
However, A professional in either field should be well versed in its counterpart. This article is meant to be advice more to programmers simply because the aim here is to improve the methodology of programming.
The ACM[#!acm!#] has formally rejected the idea of certification and formal methods of professional assessment being used in computer science. A useful quote from their executive summary is ``Such licensing practices would give false assurances of competence even if the body of knowledge were mature''[#!acm!#]. This indicates to me that the ACM believes that all computer science professionals are incompetent and there is no method for making certain otherwise. This is definitely a terrible answer from a body that should uphold the reputation and ideals of computer science. An excellent counter-argument[#!parnas!#] in parody form was written by David Parnas that points out the flaws in the ACM Council position.
Other engineering fields have professional exams and certifications that must be maintained to be certain that practitioners stay current on the state of the art and have mastered a set of skills considered to be essential. Programming needs a similar system. We should all push for genuine certification of programmers - it will add to the quality of the work in this field. Parnas[#!parnaseducation!#] writes about computer science graduates finding themselves in engineering positions and being unable to perform genuine computer program engineering tasks due to lack of training. He also lays out a program of computer program engineering that focuses on fundamentals and classical engineering principles.
We must avoid half-educated programmers from bad undergraduate programs designed to turn out java and HTML programmers who end up working on critical systems. Even worse, there are programmers who have only had a high-school education and just seem to ``be good at it''. These scenarios and how they really do play out in real life have detailed at length[#!stupid!#].
It has been claimed that certification of engineers who design buildings that people will actually occupy need to be certified but programmers do not. I wouldn't live in a building designed by an uncertified engineer any more than I would want to live in a building that was designed with software written by an incompetent programmer. Programming is becoming more and more important to society just as engineering became critical with the industrial revolution. I believe that programming must become a rigorous field and more structured just as engineering has.
Computer science is guilty of hubris in believing it that it can escape the realities that other fields have had to face. It is also guilty of having shunned the solutions other fields have used because computer science is believed to be somehow different. Computer science is slightly different in a sense because it is a marriage of engineering, science and mathematics. Each of these fields has its own set of guidelines and rules that it can bring to computer science, though. Early computer science progressed quickly because computer scientists at the time were actually electrical engineers and mathematicians who had the benefit of the rigors of training those fields provide. We've lost that recently as computer science has become viewed as something different. It may be partly due to the ease of use many computers now advertise along with the more and more simple programming languages. While the programming languages may be simple to use, correct programming is not any simpler because of them. The languages just hide the complexities. We cannot ignore the lessons that have been learned by the fields that have given rise computer science - we must add to them.
Moving to a more traditional approach to the science will be met with resistance because change is not easy. Moving from the easy-going field that computer science is now to a more defined and monitored discipline won't be well received. It is important and I believe will eventually be necessary.
The wave of claims that Open Source will make your whites whiter and your servers faster are myths promulgated by well intentioned, but overly enthusiastic, supporters of open-source.
While the media has hyped the revolutionary development process of open-source, knowledgeable and highly-regarded professionals such as Rob Pike have derided the decline and desperate situation of operating system research and new developments[#!pikesystemstalk!#]. The bursting of the bubble for business in the open-source and dot-coms area has been followed by a not widely recognized technological bust. Design and good engineering have been sacrificed for releasing code faster and more often. This ``release faster and more often'' ideology is a trademark of the open-source movement. Again, with Linux, the only innovative part of it may be the way it is developed. Linus Torvalds, leader of the Linux effort and ostensibly its chief designer, admonishes programmers to avoid design of Linux and instead it should be allowed to ``evolve''[#!linus!#]. Avoiding design is definitely a revolutionary way to develop an operating system.
This removal of testing as a part of the development process weakens the engineering being done in Open-Source projects. If a programmer (or group of programmers) knows that the next release does not have to functional because there is no penalty then there is no motivation to test. Users have become accustomed to non-working Open-Source software and so developers are not pressed to provide it with every release.
There is also no tightly connected group developing the software so there is no single plan. Several aspects of the system will be on different schedules and it's unlike that every part will be complete or stable at the same time. Scheduling the development of software projects is difficult but it's something that must be done to produce software. General open-source programmers seem to believe that software is so different from other disciplines that it cannot be planned[#!lewis!#]. Many outlets for younger programmers (such as slashdot.org[#!slashdot!#] and developer.com[#!developer!#]) grabbed onto this and took the ideas presented as far more reaching than they actually are. In short, the claim has become that programmers are artists that cannot be rushed. The attitude that software is different from other human endeavors and cannot be scheduled is the exact prima donna attitude that must be squashed. This attitude may simply be due to an ignorance of other fields and how they are similar to computer science but that is a problem in itself. Study in some of the more mature disciplines would help us greatly.
The lack of apprenticeship is a problem because one doesn't serve in the trenches and watch those who went before do the work. Valuable lessons are learned in those situations that can't be taught during 4 years in a classroom. Rather than passing on knowledge and experience from ``generation to generation'' of programmers we're leaving people to their own. They must make all the same mistakes, form the same mistaken ideas and eventually after years of working they may actually find themselves at the level of skill that the previous generation peaked at. Rather than knowledge accumulating over time, knowledge and skill must be learned from the beginning with each new group of programmers. This slows progress of the science immensely!
The flagship of open-source, Linux, is not innovative either. It's an implementation of a 30 year old system that has changed little in that time. It's already lost much of its chance of being innovative that it once had because it's now in use by industry and must grow and be shaped to be a useful tool in business rather than a research and experimentation tool for students (where it's better used).
The popular open-source license, GPL, stifles innovation. Innovation is not easy. It's not something that is cheap or quick. People, groups and companies must invest large amounts of time and money in something that may fail entirely. It's a risky exercise that few undertake. One must have a reason to risk, then. What reasons are there to risk all this? Money and fame are the main reasons. There is not much room for either when people start giving their software and innovations away.
Lately, Linux and open-source centered companies have taken to either becoming charities[#!gnome!#] or repeatedly appealing for donations in order to continue business[#!mandrake!#][#!mandrake1!#]. Software is not a charity, businesses are not altruistic entities and innovation and solid engineering cannot come from desperate organizations. The open-source philosophy and GPL-centric licenses cannot sustain the science of computing which depends on large investments of time, effort and thought that must be rewarded to continue.
Again, programmers can learn from traditional engineers. If one looks at the field that is most closely tied to programming, microprocessor manufacturing, one sees an excellent model. Hardware companies profit greatly from their innovations and that, in turn, fuels more innovation. This has set in motion a great cascade of improvements in hardware every year that is unrivaled. This is the situation that we want to establish in the software industry. Stronger protection of intellectual property (in the form of software patents), better engineering practices and a more rigorous approach to design and testing can achieve this.
Innovation is necessary for the long term survival of academic computer science and the computer software industry. It must be regained.
It also advocates integrating code with other modules very often - sometimes several times per day. This couples distinct modules so closely that modularity is often lost. It also leads to, or is a result of, poorly defined communication between modules. This tight coupling of the development of different modules also makes parallel development and testing difficult[#!ohno!#].
This is typically the kind of programming that is found in programming contests. Hastily written programs that peform a given task with little testing and nearly no planning.
There are programmers who don't understand the idea of mutual exclusion or how to synchronize processes and threads safely. They have no concept of the intricacies of locking because it has always been handled by the language.
Management of memory is another area where some programmers lack much needed skill.
The use of the word ``hack'' seems to fit into a culture of unskilled programmers. Avoid making quick and untested changes without design. Even simple changes that are obvious can lead to errors.
It's necessary to make programmers unlazy. For a culture that has involved finding shortcuts and being rewarded for being ``clever'' this is not so easy. Many disasters in engineering have involved shortcuts that turned out to have made the critical difference in some very large system. This is an obvious disaster in mechanical design (bridges, automobiles and the like) but very rarely are they as exposed and studied in programming. Exposing errors, studying them and finding where they were introduced and what problem in the development process allowed them (or caused them) will reduce the likelihood that more will appear again.
Programmers often have fragile egos so it's difficult to point out failures and mistakes without a confrontation. Removing the problem from the programmer and placing it with the system is often useful.
The best way to understand a problem is to write a thorough design for a solution but not the actual program. There's no need to write entire programs in pseudo code but a good collection of pieces include input, output and error conditions.
This idea has been documented very well elsewhere - in nearly every programming textbook, in fact. The key is to actually execute those methods.
In my experience it has been much easier to manage projects that begin with team members writing a man page, article or how-to for the system being constructed. It concisely describes what the end result should be. It is also a very useful exercise for thinking about the system in terms of the end - of the goal. Even an outline of these documents is useful. Examples can very easily become tests. If the system does not match the documentation - or the examples - then the system is flawed.
These ideas are very useful, very simple and are also very rarely taught to programmers.
Things are even worse if it doesn't fail the first time because you're less likely to go through and do it right (design your code) if it already works. ``If it ain't broke, don't fix it'' makes sense if things aren't broken. However, it is broken, you just don't recognize it!
This most often happens when one writes a piece of prototype code or proof-of-concept. This is an ok practice of that is part of the design rather than becoming the design. Trying to graft more features and extend a piece of prototype code to produce a final product is a terrible chore and usually results in a very unmaintainable piece of code.
Write scripts to do things for you. Especially when doing performance tests. The benefits of automation in testing are well known[#!programming!#]. However, it's rarely done because programmers are used to not testing methodically and waiting for the tests to complete.
``It doesn't work the same way it did yesterday'' is a way of saying ``I haven't been methodical enough''. Either you changed something and don't remember or your test didn't encapsulate all the variables in the experiment. It's possible to review a test that is implemented with scripts but it is not possible to remember something you may have forgotten already. You cannot be certain.
You are not nearly as methodical as a computer and you shouldn't try to be. The computer is a very valuable tool that does not get bored doing the same thing over and over - but humans do. Avoid the temptation to take shortcuts (even unconsciously) by avoiding boredom in the first place. Let the scripts do the boring tasks for you so that you are sure that you can reproduce your actions exactly in 5 minutes and in 5 months.
Many scientific disciplines have refined methodology for achieving reproducibility, avoiding common mistakes and making peer review easy and useful. Lab notes, lab reports, rigorous experiments and analysis are trappings of sciences that are notably absent in computer science and programming.
Above all these, guiding scientists in their work, is the scientific method. Although scientific method is very well known to most people, it might be useful to review it quickly:
These methods obviously apply to computer science. When debugging a program or trying to find the cause of a given program's behavior. Many programmers apply these methods but do so incorrectly because the application lacks scientific rigor. Many computer scientists should read [#!method!#] for an review of applying scientific method correctly.
While getting rid of software testers is probably not a good solution it does point the way to the solution. Programmers who know that software testers will catch errors are more likely to be lax and not be as thoughtful or design as carefully. Blame for a bug that slips through is moved away from tests and back to the programmers and program managers - where it is more useful. Blame being placed on testers may make better testers but don't we want better programmers (and thus better programs)? Automated regression tests, specification tests and build tests are valuable tools that have their place but they cannot be a crutch.
Programmers often sneer at testers as less skilled programmers. This isn't necessarily the case. In fact, programmers should become more skilled testers. Testing is a part of programming and should not be separated out as two independent tasks performed by different people. The loop has to be closed in order for a programmer to recognize systemic mistakes or mistakes in thinking quickly to correct the problem. A tester reporting an error through a bug-reporting system days or weeks later is not fast enough to help.
Programmers will likely not like being compared to assembly line workers but I see that there is great similarity between our profession and theirs. It is very useful to take lessons from what has worked for assembly lines and factories. Many of the ideas in this paper and the realization that computer science can gain a lot from factories came from readings of books by Taichi Ohno about improving quality and resource use in Toyota.
It is necessary for a programmer to decide how a project will end before beginning. This obvious but it is rarely actually done. Large scale beginnings, endings and milestones along the way are common and are a part of any effort but it's not a common practice for individual programmers.
Programmers can't do a perfect job all the time. In fact, a programmer shows skill in knowing when it's possible to get away with taking shortcuts when faced with real life constraints. A programmer should be at least aware of what is not being done right in order to be careful.
In an attempt to avoid the ``Ready, Fire, Aim'' problem in programming it's also important to avoid the ``Aim, aim, aim, aim...'' situation[#!stupid!#] which is nearly as common. There are times when it's necessary to actually begin writing code rather than continue design because of deadlines, time constraints and allocation of resource problems (such as available manpower!). There are ways to do this in an intelligent and planned way, though.