Hacker
From Wikipedia, the free encyclopedia
(Redirected from Computer hacking)
Jump to: navigation, search

For other uses, see Hacker (disambiguation).

Hackers are sometimes portrayed as mysterious and strange.
Enlarge
Hackers are sometimes portrayed as mysterious and strange.

A hacker is someone who creates and modifies computer software and computer hardware, including computer programming, administration, and security-related items. The term usually bears strong connotations, but may be either favorable or denigrating depending on cultural context (see the Hacker definition controversy). Common definitions include:

* In computer programming, a hacker is a software designer and programmer who builds elegant, beautiful programs and systems. A hacker can also be a programmer who hacks or reaches a goal by employing a series of modifications to exploit or extend existing code or resources. For some, "hacker" has a negative connotation and refers to a person who "hacks" or uses kludges to accomplish programming tasks that are ugly, inelegant, and inefficient. This negative form of the noun "hack" is even used among users of the positive sense of "hacker".

* In computer security, a hacker is a person who specializes in work with the security mechanisms for computer and network systems. While including those who endeavor to strengthen such mechanisms, it more often is used, especially in the mass media, to refer to those who seek access despite them.

* In other technical fields, hacker is extended to mean a person who makes things work beyond perceived limits through their own technical skill, such as a hardware hacker, or reality hacker.

* In hacker culture, a hacker is a person who has attained a certain social status and is recognized among members of the culture for commitment to the culture's values and a certain amount of technical knowledge.

Contents
[hide]

* 1 Categories of hacker
o 1.1 Hacker: Highly skilled programmer
o 1.2 Hacker: Computer and network security expert
o 1.3 Hacker: Hardware modifier
* 2 Hacker stereotypes
* 3 Recognized hackers
o 3.1 Skilled programmers
o 3.2 Security experts
o 3.3 Hacker media personalities
* 4 See also
* 5 References
* 6 Related books
* 7 External links

[edit] Categories of hacker
This section does not cite its references or sources.
You can help Wikipedia by introducing appropriate citations.

The hacker community, the set of people who would describe themselves as hackers or described by others as hackers, falls into at least four partially overlapping categories. Sometimes alternate terms such as "cracker" are used in an attempt to more exactly distinguish which category of hacker is intended, or when attempting to put a contextual distance between the categories due to the Hacker definition controversy.

[edit] Hacker: Highly skilled programmer

The positive usage of hacker is one who knows a (sometimes specified) set of programming interfaces well enough to program rapidly and expertly. This type of hacker is well-respected (although the term still carries some of the meaning of hack), and is capable of developing programs without adequate planning or where pre-planning is difficult or impossible to achieve. This zugzwang gives freedom and the ability to be creative against methodical careful progress. At their best, hackers can be very productive. The technical downside of hacker productivity is often in maintainability, documentation, and completion. Very talented hackers may become bored with a project once they have figured out all of the hard parts, and be unwilling to finish off the "details". This attitude can cause friction in environments where other programmers are expected to pick up the half finished work, decipher the structures and ideas, and bullet-proof the code. In other cases, where a hacker is willing to maintain their own code, a company may be unable to find anyone else who is capable or willing to dig through code to maintain the program if the original programmer moves on to a new job.

Additionally, there is sometimes a social downside associated with hacking. The stereotype of a hacker as having gained technical ability at a cost in social ability has historical basis in an uncomfortable amount of factual foundation in many individuals. While not universal, nor even restricted to hackers, the difficulty in relating to others and the often abrasive personalities of some hackers makes some of them difficult to work with or to organize into teams. On the other hand, it is not uncommon for hackers to thrive on social interaction.

[edit] Hacker: Computer and network security expert

Main article: Hacker (computer security)

In the networking sense, a hacker is one who specializes in work with the access control mechanisms for computer and network systems. This includes individuals who work toward maintaining and improving the integrity of such mechanisms. However, the most common usage of hacker in this respect refers to someone who exploits systems or gains unauthorized access by means of clever tactics and detailed knowledge, while taking advantage of any carelessness or ignorance on the part of system operators. This use of hacker as intruder (frequent in the media) generally has a strong negative connotation, and is disparaged and discouraged within the computer community, resulting in the modern Hacker definition controversy.

For such hackers specializing in intrusion, the highly derogatory term script kiddies is often used to indicate those who either claim to have far more skill than they actually have, or who exclusively use programs developed by others to achieve a successful security exploit.

[edit] Hacker: Hardware modifier

Main article: Hardware hacker

Another type of hacker is one who creates novel hardware modifications. At the most basic end of this spectrum are those who make frequent changes to the hardware in their computers using standard components, or make semi-cosmetic themed modifications to the appearance of the machine. This type of Hacker modifes his/her computer for performance needs and/or aesthetics. These changes often include adding memory, storage or LEDs and cold cathode tubes for light effects. These people often show off their talents in contests, and many enjoy LAN parties. At the more advanced end of the hardware hackers are those who modify hardware (not limited to computers) to expand capabilities; this group blurs into the culture of hobbyist inventors and professional electronics engineering. An example of such modification includes the addition of TCP/IP Internet capabilities to a number of vending machines and coffee makers during the late 1980s and early 1990s.

Hackers who have the ability to write circuit-level code, device drivers, firmware, low-level networking, (and even more impressively, using these techniques to make devices do things outside of their spec sheets), are typically in very high regard among hacker communities. This is primarily due to the enormous difficulty, complexity and specialized domain knowledge required for this type of work, as well as the electrical engineering expertise that plays a large role. Such hackers are rare, and almost always considered to be wizards or gurus of a very high degree.

[edit] Hacker stereotypes
This section does not cite its references or sources.
You can help Wikipedia by introducing appropriate citations.

There are theoretical types of hackers who are considered to possess an atypical level of skill beyond that of other meanings of the positive form of "hacker", which include the Guru and the Wizard.

In some portions of the computer community, a Wizard is one who can do anything a hacker can, but elegantly; while a Guru not only can do so elegantly, but instruct those who do not know how. In other sub-communities, a Guru is one with a very broad degree of expertise, while a Wizard is expert in a very narrow field. In practice, such exact distinction are usually more at home in a RPG world, and not often heard in actual conversation.

Within the mainstream media, hackers are often characterised as strange, mysterious, reclusive, and especially tricky. This may be seen as an extension of the human tendency to stigmatise what is ill-understood, which used often to be applied to natural philosophers who were often thought by superstitious neighbours to be wizards or mystics. One such example was Leonardo da Vinci, who was thought to be a necromancer due to his extensive (and extraordinary at that time) knowledge of human anatomy and his study of dead bodies.

[edit] Recognized hackers

Due to the overlapping nature of the hacker concept space, many of these individuals could be included in more than one category. See also Hacker (computer security), which has a list of people in that category, including criminal or unethical hackers.

[edit] Skilled programmers

* Tim Berners-Lee, inventor of the world wide web, while a student in the 1970s, was banned from access to the Oxford University computer for hacking access.[1]

* Dan Bernstein, the author of qmail and djbdns, also a mathematician and cryptographer.

* John Carmack, a widely recognized and influential game programmer. Through his work, he has made significant contributions to the field of 3D computer graphics and his games have sold in the millions. In 1999, Carmack appeared as number 10 in TIME's list of the 50 most influential people in technology.

* Bill Gosper, mathematician and programmer, and contemporary of Richard Greenblatt.

* Richard Greenblatt, primary designer of the MIT Lisp machine and pioneer of computerized chess.

* Grace Hopper, the first programmer of the Mark I Calculator, also developed the first compiler for a computer programming language.[2][3]

* Bill Joy, the co-founder of Sun Microsystems and author of many fundamental UNIX utilities.

* Mel Kaye, the archetypal Real Programmer[4], was credited with doing "the bulk of the programming" for the Royal McBee LGP-30 drum-memory computer in the 1950s. Ed Nather, another hacker, published the widely acclaimed "Story of Mel" in the 1980s.

* Donald Knuth, best known for practically creating the field of algorithm analysis, coding the TeX typesetting system, and writing The Art of Computer Programming - one of the most respected references in the field.

* Rasmus Lerdorf, the creator of the PHP scripting language.

* John McCarthy, the inventor of the Lisp programming language. Also coined the term "Artificial Intelligence."

* Rob Pike, a software engineer and author. He is best known for his work at Bell Labs, where he was a member of the Unix team and was involved in the creation of the Plan 9 and Inferno operating systems.

* Dennis Ritchie and Ken Thompson, who created Unix in 1969. Ritchie is also notable for having created the C programming language, from Ken Thompson's B programming language.

* Guido van Rossum, the creator of the Python programming language.

* Randal Schwartz, Perl programming language pioneer, billed himself as "Just another Perl hacker (but not what the media calls "'hacker'!)" This was in reference to characterizations made during his criminal prosecution for unauthorized computer access.

* Richard Stallman, the founder of the free software movement and the GNU project, and original author of the GPL, Emacs, GDB, and the GNU Compiler Collection. Acclaimed in Steven Levy's Hackers: Heroes of the Computer Revolution as "The Last True Hacker."

* Bjarne Stroustrup, the creator of the C++ programming language.

* Theo de Raadt, the founder of the OpenBSD project.

* Michael Tiemann, a co-founder of Cygnus Solutions, president of the Open Source Initiative. Made many contributions to the GNU C compiler, GNU debugger, and many other GNU development tools. Author of the GNU C++ compiler.

* Linus Torvalds, who was a computer science student at the University of Helsinki when he began writing the Linux kernel in 1991.

* Wietse Venema, best known for writing the Postfix mail system and co-creating (with Dan Farmer) the Security Administrator Tool for Analyzing Networks (SATAN), a remote vulnerability scanner.

* Larry Wall, the creator of the Perl programming language.[5]

* Steve Wozniak, a co-founder of Apple Computer (with Steve Jobs). Got his start making devices for phone phreaking, working with John Draper.

[edit] Security experts

* Fyodor — Author of the open source Nmap Security Scanner, web site Insecure.Org, co-author of hacking novel How to Own a Continent, and founding member of the Honeynet Project.

* Johan "Julf" Helsingius — Operated the world's most popular anonymous remailer, the Penet remailer (called penet.fi), until he closed up shop in September 1996.

* Adrian Lamo - American hacker who gained notoriety by hacking high-profile websites using common flaws in their webpages. Hacks include The New York Times, AOL, MCI Worldcom, Cingular, Google, and the NSA.

* Mark Russinovich - Expert on Windows architecture and programming; noted for identifying the limited differences between Windows NT Server and Workstation, and discovering the 2005 Sony Rootkit software

* Bruce Schneier - Founder and CTO of Counterpane Internet Security, Inc.[6]

* Phil Zimmermann - Designer and developer of PGP that brought public-key cryptography to the masses, and inadvertently upset the US government which at the time considered cryptography to be a non-exportable munition.

* Solar Designer — Founder of the Openwall Project.

[edit] Hacker media personalities

Listed below are individuals who, while fitting in one or more of the above categories, are currently more widely famous (especially among the general public) for their media presence than their technical accomplishments.

* Loyd Blankenship (also known as The Mentor) — Former LOD member. Author of The Conscience of a Hacker (Hacker's Manifesto).

* Eric Corley (also known as Emmanuel Goldstein) — Long standing publisher of 2600: The Hacker Quarterly and founder of the H.O.P.E. conferences. He has been part of the hacker community since the late '70s.

* William Henry Gates III (Bill Gates) — is the co-founder and chairman of Microsoft Corporation. Although he personally demonstrated considerable personal coding skill early in his company's history, he is most widely recognizable today as the world's richest individual. Formerly CEO, in June 2006 he also stepped down as chief software architect, with the intention of stepping down as chairman in July 2008. [1]

* Patrick K. Kroupa (also known as Lord Digital) — Former LOD member, co-founder of MindVox, author of Phantom Access programs, and MindVox: The Overture. Appears in over 20 books and hundreds of media and press articles.

* Kevin Mitnick — A former computer criminal who now (since his release from prison in 2000) speaks, consults, and authors books about social engineering and network security.

* CULT OF THE DEAD COW — A high profile hacker group that has both made news and been consulted by the media on numerous occasions.

* Eric S. Raymond — One of the founders of the Open Source Initiative. He wrote the famous text The Cathedral and the Bazaar and many other essays. He also maintains the Jargon File for the Hacker culture, which was previously maintained by Guy L. Steele, Jr..

* Bruce Perens — Also one of the founders of the Open Source Initiative. He was the former Debian GNU/Linux Project Leader, and is the primary author of the Open Source Definition.

* The 414s and Neal Patrick &mdash teenage hackers who gained brief but widespread media coverage in 1983

* Gary McKinnon &mdash accused of hacking into 97 United States military and NASA computers in 2001 and 2002. [2]

[edit] See also

* General
o Hacker culture
o Hacker ethic
o Computer crime
o Computer security
o Life hacking
o Hacker Emblem
o Hacker Manifesto
o Biohacker
o Reality hacker
o Wetware hacker
o Information wants to be free
o Video hacker

* Related
o Quick-and-dirty
o The Hacker Test
o The Snoopy Calendar program is the classic Fortran program referenced in the Hacker Test
o Hackathon

* Lists
o List of fictional hackers

[edit] References

1. ^ BBC News 31 Dec 2002 Web's inventor gets a knighthood - "Banned from using the university's computer when he and a friend were caught hacking" Accesssed October 2006.
2. ^ http://ei.cs.vt.edu/~history/Hopper.Danis.html
3. ^ http://www.hopper.navy.mil/grace/grace.htm
4. ^ http://catb.org/~esr/jargon/html/R/Real-Programmer.html
5. ^ Wall, Larry, Tom Christiansen, Jon Orwant (2000). Programming Perl, 3rd, O'Reilly. ISBN 0-596-00027-8.
6. ^ Schneier, Bruce (1994). Applied Cryptography: Protocols, Algorithms, and Source Code in C. New York: John Wiley & Sons. ISBN 0-471-59756-2.

[edit] Related books

* Levy, Steven (1984). Hackers: Heroes of the Computer Revolution. Doubleday. ISBN 0-385-19195-2.
* Sterling, Bruce (1992). The Hacker Crackdown. Bantam. ISBN 0-553-08058-X.
* Slatalla, Michelle, Joshua Quittner (1995). Masters of Deception: The Gang That Ruled Cyberspace. HarperCollins. ISBN 0-06-017030-1.
* Dreyfus, Suelette (1997). Underground: Tales of Hacking, Madness and Obsession on the Electronic Frontier. Mandarin. ISBN 1-86330-595-5.
* Verton, Dan (2002). The Hacker Diaries : Confessions of Teenage Hackers. McGraw-Hill Osborne Media. ISBN 0-07-222364-2.

[edit] External links
Find more information on Hacker by searching Wikipedia's sister projects:

Dictionary definitions from Wiktionary
Textbooks from Wikibooks
Quotations from Wikiquote
Source texts from Wikisource
Images and media from Commons
News stories from Wikinews

* The Jargon File
* The New Jargon File: An effort to establish an open hacker culture document, in the tradition of the Jargon File
* How To Become A Hacker by Eric S. Raymond
* On Hacking by Richard M. Stallman
* The MIT Gallery of Hacks




How To Become A Hacker
Eric Steven Raymond
Thyrsus Enterprises





Copyright © 2001 Eric S. Raymond
Revision History
Revision 1.35 03 Aug 2006 esr
Minor fixes.
Revision 1.34 07 Mar 2006 esr
Remove C# from the list of languages to be avoided now that Mono is out of beta.
Revision 1.33 29 Nov 2005 esr
Add a pointer to Peter Norvig's excellent essay.
Revision 1.32 29 Jun 2005 esr
Substantial new material on not solving problems twice. Answer a FAQ on hacking and open-source programming. The three questions that reveal if you are already a hacker.
Revision 1.31 22 Mar 2005 esr
Added a link to another Paul Graham essay, and advice on how to pick a first project. More translation-link updates.
Revision 1.30 2 Mar 2005 esr
Added and updated many translation links.

Table of Contents

Why This Document?
What Is a Hacker?
The Hacker Attitude

1. The world is full of fascinating problems waiting to be solved.
2. No problem should ever have to be solved twice.
3. Boredom and drudgery are evil.
4. Freedom is good.
5. Attitude is no substitute for competence.

Basic Hacking Skills

1. Learn how to program.
2. Get one of the open-source Unixes and learn to use and run it.
3. Learn how to use the World Wide Web and write HTML.
4. If you don't have functional English, learn it.

Status in the Hacker Culture

1. Write open-source software
2. Help test and debug open-source software
3. Publish useful information
4. Help keep the infrastructure working
5. Serve the hacker culture itself

The Hacker/Nerd Connection
Points For Style
Other Resources
Frequently Asked Questions

Why This Document?

As editor of the Jargon File and author of a few other well-known documents of similar nature, I often get email requests from enthusiastic network newbies asking (in effect) "how can I learn to be a wizardly hacker?". Back in 1996 I noticed that there didn't seem to be any other FAQs or web documents that addressed this vital question, so I started this one. A lot of hackers now consider it definitive, and I suppose that means it is. Still, I don't claim to be the exclusive authority on this topic; if you don't like what you read here, write your own.

If you are reading a snapshot of this document offline, the current version lives at http://catb.org/~esr/faqs/hacker-howto.html.

Note: there is a list of Frequently Asked Questions at the end of this document. Please read these—twice—before mailing me any questions about this document.

Numerous translations of this document are available: Arabic Bulgarian, Catalan, Chinese (Simplified), Danish, Dutch, Farsi, Finnish, German, Greek Hebrew, Italian Japanese, Norwegian, Polish, Portuguese (Brazilian), Romanian Russian Spanish, Turkish, and Swedish. Note that since this document changes occasionally, they may be out of date to varying degrees.

The five-dots-in-nine-squares diagram that decorates this document is called a glider. It is a simple pattern with some surprising properties in a mathematical simulation called Life that has fascinated hackers for many years. I think it makes a good visual emblem for what hackers are like — abstract, at first a bit mysterious-seeming, but a gateway to a whole world with an intricate logic of its own. Read more about the glider emblem here.
What Is a Hacker?

The Jargon File contains a bunch of definitions of the term ‘hacker’, most having to do with technical adeptness and a delight in solving problems and overcoming limits. If you want to know how to become a hacker, though, only two are really relevant.

There is a community, a shared culture, of expert programmers and networking wizards that traces its history back through decades to the first time-sharing minicomputers and the earliest ARPAnet experiments. The members of this culture originated the term ‘hacker’. Hackers built the Internet. Hackers made the Unix operating system what it is today. Hackers run Usenet. Hackers make the World Wide Web work. If you are part of this culture, if you have contributed to it and other people in it know who you are and call you a hacker, you're a hacker.

The hacker mind-set is not confined to this software-hacker culture. There are people who apply the hacker attitude to other things, like electronics or music — actually, you can find it at the highest levels of any science or art. Software hackers recognize these kindred spirits elsewhere and may call them ‘hackers’ too — and some claim that the hacker nature is really independent of the particular medium the hacker works in. But in the rest of this document we will focus on the skills and attitudes of software hackers, and the traditions of the shared culture that originated the term ‘hacker’.

There is another group of people who loudly call themselves hackers, but aren't. These are people (mainly adolescent males) who get a kick out of breaking into computers and phreaking the phone system. Real hackers call these people ‘crackers’ and want nothing to do with them. Real hackers mostly think crackers are lazy, irresponsible, and not very bright, and object that being able to break security doesn't make you a hacker any more than being able to hotwire cars makes you an automotive engineer. Unfortunately, many journalists and writers have been fooled into using the word ‘hacker’ to describe crackers; this irritates real hackers no end.

The basic difference is this: hackers build things, crackers break them.

If you want to be a hacker, keep reading. If you want to be a cracker, go read the alt.2600 newsgroup and get ready to do five to ten in the slammer after finding out you aren't as smart as you think you are. And that's all I'm going to say about crackers.
The Hacker Attitude

1. The world is full of fascinating problems waiting to be solved.
2. No problem should ever have to be solved twice.
3. Boredom and drudgery are evil.
4. Freedom is good.
5. Attitude is no substitute for competence.

Hackers solve problems and build things, and they believe in freedom and voluntary mutual help. To be accepted as a hacker, you have to behave as though you have this kind of attitude yourself. And to behave as though you have the attitude, you have to really believe the attitude.

But if you think of cultivating hacker attitudes as just a way to gain acceptance in the culture, you'll miss the point. Becoming the kind of person who believes these things is important for you — for helping you learn and keeping you motivated. As with all creative arts, the most effective way to become a master is to imitate the mind-set of masters — not just intellectually but emotionally as well.

Or, as the following modern Zen poem has it:


To follow the path:
look to the master,
follow the master,
walk with the master,
see through the master,
become the master.

So, if you want to be a hacker, repeat the following things until you believe them:
1. The world is full of fascinating problems waiting to be solved.

Being a hacker is lots of fun, but it's a kind of fun that takes lots of effort. The effort takes motivation. Successful athletes get their motivation from a kind of physical delight in making their bodies perform, in pushing themselves past their own physical limits. Similarly, to be a hacker you have to get a basic thrill from solving problems, sharpening your skills, and exercising your intelligence.

If you aren't the kind of person that feels this way naturally, you'll need to become one in order to make it as a hacker. Otherwise you'll find your hacking energy is sapped by distractions like sex, money, and social approval.

(You also have to develop a kind of faith in your own learning capacity — a belief that even though you may not know all of what you need to solve a problem, if you tackle just a piece of it and learn from that, you'll learn enough to solve the next piece — and so on, until you're done.)
2. No problem should ever have to be solved twice.

Creative brains are a valuable, limited resource. They shouldn't be wasted on re-inventing the wheel when there are so many fascinating new problems waiting out there.

To behave like a hacker, you have to believe that the thinking time of other hackers is precious — so much so that it's almost a moral duty for you to share information, solve problems and then give the solutions away just so other hackers can solve new problems instead of having to perpetually re-address old ones.

Note, however, that "No problem should ever have to be solved twice." does not imply that you have to consider all existing solutions sacred, or that there is only one right solution to any given problem. Often, we learn a lot about the problem that we didn't know before by studying the first cut at a solution. It's OK, and often necessary, to decide that we can do better. What's not OK is artificial technical, legal, or institutional barriers (like closed-source code) that prevent a good solution from being re-used and force people to re-invent wheels.

(You don't have to believe that you're obligated to give all your creative product away, though the hackers that do are the ones that get most respect from other hackers. It's consistent with hacker values to sell enough of it to keep you in food and rent and computers. It's fine to use your hacking skills to support a family or even get rich, as long as you don't forget your loyalty to your art and your fellow hackers while doing it.)
3. Boredom and drudgery are evil.

Hackers (and creative people in general) should never be bored or have to drudge at stupid repetitive work, because when this happens it means they aren't doing what only they can do — solve new problems. This wastefulness hurts everybody. Therefore boredom and drudgery are not just unpleasant but actually evil.

To behave like a hacker, you have to believe this enough to want to automate away the boring bits as much as possible, not just for yourself but for everybody else (especially other hackers).

(There is one apparent exception to this. Hackers will sometimes do things that may seem repetitive or boring to an observer as a mind-clearing exercise, or in order to acquire a skill or have some particular kind of experience you can't have otherwise. But this is by choice — nobody who can think should ever be forced into a situation that bores them.)
4. Freedom is good.

Hackers are naturally anti-authoritarian. Anyone who can give you orders can stop you from solving whatever problem you're being fascinated by — and, given the way authoritarian minds work, will generally find some appallingly stupid reason to do so. So the authoritarian attitude has to be fought wherever you find it, lest it smother you and other hackers.

(This isn't the same as fighting all authority. Children need to be guided and criminals restrained. A hacker may agree to accept some kinds of authority in order to get something he wants more than the time he spends following orders. But that's a limited, conscious bargain; the kind of personal surrender authoritarians want is not on offer.)

Authoritarians thrive on censorship and secrecy. And they distrust voluntary cooperation and information-sharing — they only like ‘cooperation’ that they control. So to behave like a hacker, you have to develop an instinctive hostility to censorship, secrecy, and the use of force or deception to compel responsible adults. And you have to be willing to act on that belief.
5. Attitude is no substitute for competence.

To be a hacker, you have to develop some of these attitudes. But copping an attitude alone won't make you a hacker, any more than it will make you a champion athlete or a rock star. Becoming a hacker will take intelligence, practice, dedication, and hard work.

Therefore, you have to learn to distrust attitude and respect competence of every kind. Hackers won't let posers waste their time, but they worship competence — especially competence at hacking, but competence at anything is valued. Competence at demanding skills that few can master is especially good, and competence at demanding skills that involve mental acuteness, craft, and concentration is best.

If you revere competence, you'll enjoy developing it in yourself — the hard work and dedication will become a kind of intense play rather than drudgery. That attitude is vital to becoming a hacker.
Basic Hacking Skills

1. Learn how to program.
2. Get one of the open-source Unixes and learn to use and run it.
3. Learn how to use the World Wide Web and write HTML.
4. If you don't have functional English, learn it.

The hacker attitude is vital, but skills are even more vital. Attitude is no substitute for competence, and there's a certain basic toolkit of skills which you have to have before any hacker will dream of calling you one.

This toolkit changes slowly over time as technology creates new skills and makes old ones obsolete. For example, it used to include programming in machine language, and didn't until recently involve HTML. But right now it pretty clearly includes the following:
1. Learn how to program.

This, of course, is the fundamental hacking skill. If you don't know any computer languages, I recommend starting with Python. It is cleanly designed, well documented, and relatively kind to beginners. Despite being a good first language, it is not just a toy; it is very powerful and flexible and well suited for large projects. I have written a more detailed evaluation of Python. Good tutorials are available at the Python web site.

Java is also a good language for learning to program in. It is more difficult than Python, but produces faster code than Python. I think it makes an excellent second language. Unfortunately, Sun's reference implementation is still proprietary. This is not so much an issue with the Java language itself, as high-quality open-source Java interpreters are readily available; the real problem is the class libraries that travel with the language. The open-source class libraries lag behind Sun's. So, if you do choose to learn Java, do it with one of the open-source implementations rather than becoming dependent on Sun's proprietary code.

But be aware that you won't reach the skill level of a hacker or even merely a programmer if you only know one or two languages — you need to learn how to think about programming problems in a general way, independent of any one language. To be a real hacker, you need to get to the point where you can learn a new language in days by relating what's in the manual to what you already know. This means you should learn several very different languages.

If you get into serious programming, you will have to learn C, the core language of Unix. C++ is very closely related to C; if you know one, learning the other will not be difficult. Neither language is a good one to try learning as your first, however. And, actually, the more you can avoid programming in C the more productive you will be.

C is very efficient, and very sparing of your machine's resources. Unfortunately, C gets that efficiency by requiring you to do a lot of low-level management of resources (like memory) by hand. All that low-level code is complex and bug-prone, and will soak up huge amounts of your time on debugging. With today's machines as powerful as they are, this is usually a bad tradeoff — it's smarter to use a language that uses the machine's time less efficiently, but your time much more efficiently. Thus, Python.

Other languages of particular importance to hackers include Perl and LISP. Perl is worth learning for practical reasons; it's very widely used for active web pages and system administration, so that even if you never write Perl you should learn to read it. Many people use Perl in the way I suggest you should use Python, to avoid C programming on jobs that don't require C's machine efficiency. You will need to be able to understand their code.

LISP is worth learning for a different reason — the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use LISP itself a lot. (You can get some beginning experience with LISP fairly easily by writing and modifying editing modes for the Emacs text editor, or Script-Fu plugins for the GIMP.)

It's best, actually, to learn all five of Python, C/C++, Java, Perl, and LISP. Besides being the most important hacking languages, they represent very different approaches to programming, and each will educate you in valuable ways.

I can't give complete instructions on how to learn to program here — it's a complex skill. But I can tell you that books and courses won't do it (many, maybe most of the best hackers are self-taught). You can learn language features — bits of knowledge — from books, but the mind-set that makes that knowledge into living skill can be learned only by practice and apprenticeship. What will do it is (a) reading code and (b) writing code.

Peter Norvig, who is one of Google's top hackers and the co-author of the most widely used textbook on AI, has written an excellent essay called Teach Yourself Programming in Ten Years. His "recipe for programming success" is worth careful attention.

Learning to program is like learning to write good natural language. The best way to do it is to read some stuff written by masters of the form, write some things yourself, read a lot more, write a little more, read a lot more, write some more ... and repeat until your writing begins to develop the kind of strength and economy you see in your models.

Finding good code to read used to be hard, because there were few large programs available in source for fledgeling hackers to read and tinker with. This has changed dramatically; open-source software, programming tools, and operating systems (all built by hackers) are now widely available. Which brings me neatly to our next topic...
2. Get one of the open-source Unixes and learn to use and run it.

I'll assume you have a personal computer or can get access to one. (Take a moment to appreciate how much that means. The hacker culture originally evolved back when computers were so expensive that individuals could not own them.) The single most important step any newbie can take toward acquiring hacker skills is to get a copy of Linux or one of the BSD-Unixes or OpenSolaris, install it on a personal machine, and run it.

Yes, there are other operating systems in the world besides Unix. But they're distributed in binary — you can't read the code, and you can't modify it. Trying to learn to hack on a Microsoft Windows machine or under any other closed-source system is like trying to learn to dance while wearing a body cast.

Under Mac OS X it's possible, but only part of the system is open source — you're likely to hit a lot of walls, and you have to be careful not to develop the bad habit of depending on Apple's proprietary code. If you concentrate on the Unix under the hood you can learn some useful things.

Unix is the operating system of the Internet. While you can learn to use the Internet without knowing Unix, you can't be an Internet hacker without understanding Unix. For this reason, the hacker culture today is pretty strongly Unix-centered. (This wasn't always true, and some old-time hackers still aren't happy about it, but the symbiosis between Unix and the Internet has become strong enough that even Microsoft's muscle doesn't seem able to seriously dent it.)

So, bring up a Unix — I like Linux myself but there are other ways (and yes, you can run both Linux and Microsoft Windows on the same machine). Learn it. Run it. Tinker with it. Talk to the Internet with it. Read the code. Modify the code. You'll get better programming tools (including C, LISP, Python, and Perl) than any Microsoft operating system can dream of hosting, you'll have fun, and you'll soak up more knowledge than you realize you're learning until you look back on it as a master hacker.

For more about learning Unix, see The Loginataka. You might also want to have a look at The Art Of Unix Programming.

To get your hands on a Linux, see the Linux Online! site; you can download from there or (better idea) find a local Linux user group to help you with installation. From a new user's point of view, all Linux distributions are pretty much equivalent.

You can find BSD Unix help and resources at www.bsd.org.

I have written a primer on the basics of Unix and the Internet.

(Note: I don't really recommend installing either Linux or BSD as a solo project if you're a newbie. For Linux, find a local Linux user's group and ask for help.)
3. Learn how to use the World Wide Web and write HTML.

Most of the things the hacker culture has built do their work out of sight, helping run factories and offices and universities without any obvious impact on how non-hackers live. The Web is the one big exception, the huge shiny hacker toy that even politicians admit has changed the world. For this reason alone (and a lot of other good ones as well) you need to learn how to work the Web.

This doesn't just mean learning how to drive a browser (anyone can do that), but learning how to write HTML, the Web's markup language. If you don't know how to program, writing HTML will teach you some mental habits that will help you learn. So build a home page. Try to stick to XHTML, which is a cleaner language than classic HTML. (There are good beginner tutorials on the Web; here's one.)

But just having a home page isn't anywhere near good enough to make you a hacker. The Web is full of home pages. Most of them are pointless, zero-content sludge — very snazzy-looking sludge, mind you, but sludge all the same (for more on this see The HTML Hell Page).

To be worthwhile, your page must have content — it must be interesting and/or useful to other hackers. And that brings us to the next topic...
4. If you don't have functional English, learn it.

As an American and native English-speaker myself, I have previously been reluctant to suggest this, lest it be taken as a sort of cultural imperialism. But several native speakers of other languages have urged me to point out that English is the working language of the hacker culture and the Internet, and that you will need to know it to function in the hacker community.

Back around 1991 I learned that many hackers who have English as a second language use it in technical discussions even when they share a birth tongue; it was reported to me at the time that English has a richer technical vocabulary than any other language and is therefore simply a better tool for the job. For similar reasons, translations of technical books written in English are often unsatisfactory (when they get done at all).

Linus Torvalds, a Finn, comments his code in English (it apparently never occurred to him to do otherwise). His fluency in English has been an important factor in his ability to recruit a worldwide community of developers for Linux. It's an example worth following.

Being a native English-speaker does not guarantee that you have language skills good enough to function as a hacker. If your writing is semi-literate, ungrammatical, and riddled with misspellings, many hackers (including myself) will tend to ignore you. While sloppy writing does not invariably mean sloppy thinking, we've generally found the correlation to be strong — and we have no use for sloppy thinkers. If you can't yet write competently, learn to.
Status in the Hacker Culture

1. Write open-source software
2. Help test and debug open-source software
3. Publish useful information
4. Help keep the infrastructure working
5. Serve the hacker culture itself

Like most cultures without a money economy, hackerdom runs on reputation. You're trying to solve interesting problems, but how interesting they are, and whether your solutions are really good, is something that only your technical peers or superiors are normally equipped to judge.

Accordingly, when you play the hacker game, you learn to keep score primarily by what other hackers think of your skill (this is why you aren't really a hacker until other hackers consistently call you one). This fact is obscured by the image of hacking as solitary work; also by a hacker-cultural taboo (gradually decaying since the late 1990s but still potent) against admitting that ego or external validation are involved in one's motivation at all.

Specifically, hackerdom is what anthropologists call a gift culture. You gain status and reputation in it not by dominating other people, nor by being beautiful, nor by having things other people want, but rather by giving things away. Specifically, by giving away your time, your creativity, and the results of your skill.

There are basically five kinds of things you can do to be respected by hackers:
1. Write open-source software

The first (the most central and most traditional) is to write programs that other hackers think are fun or useful, and give the program sources away to the whole hacker culture to use.

(We used to call these works “free software”, but this confused too many people who weren't sure exactly what “free” was supposed to mean. Most of us now prefer the term “open-source” software).

Hackerdom's most revered demigods are people who have written large, capable programs that met a widespread need and given them away, so that now everyone uses them.

But there's a bit of a fine historical point here. While hackers have always looked up to the open-source developers among them as our community's hardest core, before the mid-1990s most hackers most of the time worked on closed source. This was still true when I wrote the first version of this HOWTO in 1996; it took the mainstreaming of open-source software after 1997 to change things. Today, "the hacker community" and "open-source developers" are two descriptions for what is essentially the same culture and population — but it is worth remembering that this was not always so.
2. Help test and debug open-source software

They also serve who stand and debug open-source software. In this imperfect world, we will inevitably spend most of our software development time in the debugging phase. That's why any open-source author who's thinking will tell you that good beta-testers (who know how to describe symptoms clearly, localize problems well, can tolerate bugs in a quickie release, and are willing to apply a few simple diagnostic routines) are worth their weight in rubies. Even one of these can make the difference between a debugging phase that's a protracted, exhausting nightmare and one that's merely a salutary nuisance.

If you're a newbie, try to find a program under development that you're interested in and be a good beta-tester. There's a natural progression from helping test programs to helping debug them to helping modify them. You'll learn a lot this way, and generate good karma with people who will help you later on.
3. Publish useful information

Another good thing is to collect and filter useful and interesting information into web pages or documents like Frequently Asked Questions (FAQ) lists, and make those generally available.

Maintainers of major technical FAQs get almost as much respect as open-source authors.
4. Help keep the infrastructure working

The hacker culture (and the engineering development of the Internet, for that matter) is run by volunteers. There's a lot of necessary but unglamorous work that needs done to keep it going — administering mailing lists, moderating newsgroups, maintaining large software archive sites, developing RFCs and other technical standards.

People who do this sort of thing well get a lot of respect, because everybody knows these jobs are huge time sinks and not as much fun as playing with code. Doing them shows dedication.
5. Serve the hacker culture itself

Finally, you can serve and propagate the culture itself (by, for example, writing an accurate primer on how to become a hacker :-)). This is not something you'll be positioned to do until you've been around for while and become well-known for one of the first four things.

The hacker culture doesn't have leaders, exactly, but it does have culture heroes and tribal elders and historians and spokespeople. When you've been in the trenches long enough, you may grow into one of these. Beware: hackers distrust blatant ego in their tribal elders, so visibly reaching for this kind of fame is dangerous. Rather than striving for it, you have to sort of position yourself so it drops in your lap, and then be modest and gracious about your status.
The Hacker/Nerd Connection

Contrary to popular myth, you don't have to be a nerd to be a hacker. It does help, however, and many hackers are in fact nerds. Being something a social outcast helps you stay concentrated on the really important things, like thinking and hacking.

For this reason, many hackers have adopted the label ‘geek’ as a badge of pride — it's a way of declaring their independence from normal social expectations (as well as a fondness for other things like science fiction and strategy games that often go with being a hacker). The term 'nerd' used to be used this way back in the 1990s, back when 'nerd' was a mild pejorative and 'geek' a rather harsher one; sometime after 2000 they switched places, at least in U.S. popular culture, and there is now even a significant geek-pride culture among people who aren't techies.

If you can manage to concentrate enough on hacking to be good at it and still have a life, that's fine. This is a lot easier today than it was when I was a newbie in the 1970s; mainstream culture is much friendlier to techno-nerds now. There are even growing numbers of people who realize that hackers are often high-quality lover and spouse material.

If you're attracted to hacking because you don't have a life, that's OK too — at least you won't have trouble concentrating. Maybe you'll get a life later on.
Points For Style

Again, to be a hacker, you have to enter the hacker mindset. There are some things you can do when you're not at a computer that seem to help. They're not substitutes for hacking (nothing is) but many hackers do them, and feel that they connect in some basic way with the essence of hacking.

*

Learn to write your native language well. Though it's a common stereotype that programmers can't write, a surprising number of hackers (including all the most accomplished ones I know of) are very able writers.
*

Read science fiction. Go to science fiction conventions (a good way to meet hackers and proto-hackers).
*

Train in a martial-arts form. The kind of mental discipline required for martial arts seems to be similar in important ways to what hackers do. The most popular forms among hackers are definitely Asian empty-hand arts such as Tae Kwon Do, various forms of Karate, Kung Fu, Aikido, or Ju Jitsu. Western fencing and Asian sword arts also have visible followings. In places where it's legal, pistol shooting has been rising in popularity since the late 1990s. The most hackerly martial arts are those which emphasize mental discipline, relaxed awareness, and control, rather than raw strength, athleticism, or physical toughness.
*

Study an actual meditation discipline. The perennial favorite among hackers is Zen (importantly, it is possible to benefit from Zen without acquiring a religion or discarding one you already have). Other styles may work as well, but be careful to choose one that doesn't require you to believe crazy things.
*

Develop an analytical ear for music. Learn to appreciate peculiar kinds of music. Learn to play some musical instrument well, or how to sing.
*

Develop your appreciation of puns and wordplay.

The more of these things you already do, the more likely it is that you are natural hacker material. Why these things in particular is not completely clear, but they're connected with a mix of left- and right-brain skills that seems to be important; hackers need to be able to both reason logically and step outside the apparent logic of a problem at a moment's notice.

Work as intensely as you play and play as intensely as you work. For true hackers, the boundaries between "play", "work", "science" and "art" all tend to disappear, or to merge into a high-level creative playfulness. Also, don't be content with a narrow range of skills. Though most hackers self-describe as programmers, they are very likely to be more than competent in several related skills — system administration, web design, and PC hardware troubleshooting are common ones. A hacker who's a system administrator, on the other hand, is likely to be quite skilled at script programming and web design. Hackers don't do things by halves; if they invest in a skill at all, they tend to get very good at it.

Finally, a few things not to do.

*

Don't use a silly, grandiose user ID or screen name.
*

Don't get in flame wars on Usenet (or anywhere else).
*

Don't call yourself a ‘cyberpunk’, and don't waste your time on anybody who does.
*

Don't post or email writing that's full of spelling errors and bad grammar.

The only reputation you'll make doing any of these things is as a twit. Hackers have long memories — it could take you years to live your early blunders down enough to be accepted.

The problem with screen names or handles deserves some amplification. Concealing your identity behind a handle is a juvenile and silly behavior characteristic of crackers, warez d00dz, and other lower life forms. Hackers don't do this; they're proud of what they do and want it associated with their real names. So if you have a handle, drop it. In the hacker culture it will only mark you as a loser.
Other Resources

Paul Graham has written an essay called Great Hackers, and another on Undergraduation, in which he speaks much wisdom.

Peter Seebach maintains an excellent Hacker FAQ for managers who don't understand how to deal with hackers.

There is a document called How To Be A Programmer that is an excellent complement to this one. It has valuable advice not just about coding and skillsets, but about how to function on a programming team.

I have also written A Brief History Of Hackerdom.

I have written a paper, The Cathedral and the Bazaar, which explains a lot about how the Linux and open-source cultures work. I have addressed this topic even more directly in its sequel Homesteading the Noosphere.

Rick Moen has written an excellent document on how to run a Linux user group.

Rick Moen and I have collaborated on another document on How To Ask Smart Questions. This will help you seek assistance in a way that makes it more likely that you will actually get it.

If you need instruction in the basics of how personal computers, Unix, and the Internet work, see The Unix and Internet Fundamentals HOWTO.

When you release software or write patches for software, try to follow the guidelines in the Software Release Practice HOWTO.

If you enjoyed the Zen poem, you might also like Rootless Root: The Unix Koans of Master Foo.
Frequently Asked Questions

Q: How do I tell if I am already a hacker?
Q: Will you teach me how to hack?
Q: How can I get started, then?
Q: When do you have to start? Is it too late for me to learn?
Q: How long will it take me to learn to hack?
Q: Is Visual Basic a good language to start with?
Q: Would you help me to crack a system, or teach me how to crack?
Q: How can I get the password for someone else's account?
Q: How can I break into/read/monitor someone else's email?
Q: How can I steal channel op privileges on IRC?
Q: I've been cracked. Will you help me fend off further attacks?
Q: I'm having problems with my Windows software. Will you help me?
Q: Where can I find some real hackers to talk with?
Q: Can you recommend useful books about hacking-related subjects?
Q: Do I need to be good at math to become a hacker?
Q: What language should I learn first?
Q: What kind of hardware do I need?
Q: I want to contribute. Can you help me pick a problem to work on?
Q: Do I need to hate and bash Microsoft?
Q: But won't open-source software leave programmers unable to make a living?
Q: Where can I get a free Unix?

Q:

How do I tell if I am already a hacker?
A:

Ask yourself the following three questions:

*

Do you speak code, fluently?
*

Do you identify with the goals and values of the hacker community?
*

Has a well-established member of the hacker community ever called you a hacker?

If you can answer yes to all three of these questions, you are already a hacker. No two alone are sufficient.

The first test is about skills. You probably pass it if you have the minimum technical skills described earlier in this document. You blow right through it if you have had a substantial amount of code accepted by an open-source development project.

The second test is about attitude. If the five principles of the hacker mindset seemed obvious to you, more like a description of the way you already live than anything novel, you are already halfway to passing it. That's the inward half; the other, outward half is the degree to which you identify with the hacker community's long-term projects.

Here is an incomplete but indicative list of some of those projects: Does it matter to you that Linux improve and spread? Are you passionate about software freedom? Hostile to monopolies? Do you act on the belief that computers can be instruments of empowerment that make the world a richer and more humane place?

But a note of caution is in order here. The hacker community has some specific, primarily defensive political interests — two of them are defending free-speech rights and fending off "intellectual-property" power grabs that would make open source illegal. Some of those long-term projects are civil-liberties organizations like the Electronic Frontier Foundation, and the outward attitude properly includes support of them. But beyond that, most hackers view attempts to systematize the hacker attitude into an explicit political program with suspicion; we've learned, the hard way, that these attempts are divisive and distracting. If someone tries to recruit you to march on your capitol in the name of the hacker attitude, they've missed the point. The right response is probably “Shut up and show them the code.”

The third test has a tricky element of recursiveness about it. I observed in the section called “What Is a Hacker?” that being a hacker is partly a matter of belonging to a particular subculture or social network with a shared history, an inside and an outside. In the far past, hackers were a much less cohesive and self-aware group than they are today. But the importance of the social-network aspect has increased over the last thirty years as the Internet has made connections with the core of the hacker subculture easier to develop and maintain. One easy behavioral index of the change is that, in this century, we have our own T-shirts.

Sociologists, who study networks like those of the hacker culture under the general rubric of "invisible colleges", have noted that one characteristic of such networks is that they have gatekeepers — core members with the social authority to endorse new members into the network. Because the "invisible college" that is hacker culture is a loose and informal one, the role of gatekeeper is informal too. But one thing that all hackers understand in their bones is that not every hacker is a gatekeeper. Gatekeepers have to have a certain degree of seniority and accomplishment before they can bestow the title. How much is hard to quantify, but every hacker knows it when they see it.
Q:

Will you teach me how to hack?
A:

Since first publishing this page, I've gotten several requests a week (often several a day) from people to "teach me all about hacking". Unfortunately, I don't have the time or energy to do this; my own hacking projects, and working as an open-source advocate, take up 110% of my time.

Even if I did, hacking is an attitude and skill you basically have to teach yourself. You'll find that while real hackers want to help you, they won't respect you if you beg to be spoon-fed everything they know.

Learn a few things first. Show that you're trying, that you're capable of learning on your own. Then go to the hackers you meet with specific questions.

If you do email a hacker asking for advice, here are two things to know up front. First, we've found that people who are lazy or careless in their writing are usually too lazy and careless in their thinking to make good hackers — so take care to spell correctly, and use good grammar and punctuation, otherwise you'll probably be ignored. Secondly, don't dare ask for a reply to an ISP account that's different from the account you're sending from; we find people who do that are usually thieves using stolen accounts, and we have no interest in rewarding or assisting thievery.
Q:

How can I get started, then?
A:

The best way for you to get started would probably be to go to a LUG (Linux user group) meeting. You can find such groups on the LDP General Linux Information Page; there is probably one near you, possibly associated with a college or university. LUG members will probably give you a Linux if you ask, and will certainly help you install one and get started.
Q:

When do you have to start? Is it too late for me to learn?
A:

Any age at which you are motivated to start is a good age. Most people seem to get interested between ages 15 and 20, but I know of exceptions in both directions.
Q:

How long will it take me to learn to hack?
A:

That depends on how talented you are and how hard you work at it. Most people can acquire a respectable skill set in eighteen months to two years, if they concentrate. Don't think it ends there, though; if you are a real hacker, you will spend the rest of your life learning and perfecting your craft.
Q:

Is Visual Basic a good language to start with?
A:

If you're asking this question, it almost certainly means you're thinking about trying to hack under Microsoft Windows. This is a bad idea in itself. When I compared trying to learn to hack under Windows to trying to learn to dance while wearing a body cast, I wasn't kidding. Don't go there. It's ugly, and it never stops being ugly.

There is a specific problem with Visual Basic; mainly that it's not portable. Though there is a prototype open-source implementations of Visual Basic, the applicable ECMA standards don't cover more than a small set of its programming interfaces. On Windows most of its library support is proprietary to a single vendor (Microsoft); if you aren't extremely careful about which features you use — more careful than any newbie is really capable of being — you'll end up locked into only those platforms Microsoft chooses to support. If you're starting on a Unix, much better languages with better libraries are available. Python, for example.

Also, like other Basics, Visual Basic is a poorly-designed language that will teach you bad programming habits. No, don't ask me to describe them in detail; that explanation would fill a book. Learn a well-designed language instead.

One of those bad habits is becoming dependent on a single vendor's libraries, widgets, and development tools. In general, any language that isn't fully supported under at least Linux or one of the BSDs, and/or at least three different vendors' operating systems, is a poor one to learn to hack in.
Q:

Would you help me to crack a system, or teach me how to crack?
A:

No. Anyone who can still ask such a question after reading this FAQ is too stupid to be educable even if I had the time for tutoring. Any emailed requests of this kind that I get will be ignored or answered with extreme rudeness.
Q:

How can I get the password for someone else's account?
A:

This is cracking. Go away, idiot.
Q:

How can I break into/read/monitor someone else's email?
A:

This is cracking. Get lost, moron.
Q:

How can I steal channel op privileges on IRC?
A:

This is cracking. Begone, cretin.
Q:

I've been cracked. Will you help me fend off further attacks?
A:

No. Every time I've been asked this question so far, it's been from some poor sap running Microsoft Windows. It is not possible to effectively secure Windows systems against crack attacks; the code and architecture simply have too many flaws, which makes securing Windows like trying to bail out a boat with a sieve. The only reliable prevention starts with switching to Linux or some other operating system that is designed to at least be capable of security.
Q:

I'm having problems with my Windows software. Will you help me?
A:

Yes. Go to a DOS prompt and type "format c:". Any problems you are experiencing will cease within a few minutes.
Q:

Where can I find some real hackers to talk with?
A:

The best way is to find a Unix or Linux user's group local to you and go to their meetings (you can find links to several lists of user groups on the LDP site at ibiblio).

(I used to say here that you wouldn't find any real hackers on IRC, but I'm given to understand this is changing. Apparently some real hacker communities, attached to things like GIMP and Perl, have IRC channels now.)
Q:

Can you recommend useful books about hacking-related subjects?
A:

I maintain a Linux Reading List HOWTO that you may find helpful. The Loginataka may also be interesting.

For an introduction to Python, see the introductory materials on the Python site.
Q:

Do I need to be good at math to become a hacker?
A:

No. Hacking uses very little formal mathematics or arithmetic. In particular, you won't usually need trigonometry, calculus or analysis (there are exceptions to this in a handful of specific application areas like 3-D computer graphics). Knowing some formal logic and Boolean algebra is good. Some grounding in finite mathematics (including finite-set theory, combinatorics, and graph theory) can be helpful.

Much more importantly: you need to be able to think logically and follow chains of exact reasoning, the way mathematicians do. While the content of most mathematics won't help you, you will need the discipline and intelligence to handle mathematics. If you lack the intelligence, there is little hope for you as a hacker; if you lack the discipline, you'd better grow it.

I think a good way to find out if you have what it takes is to pick up a copy of Raymond Smullyan's book What Is The Name Of This Book?. Smullyan's playful logical conundrums are very much in the hacker spirit. Being able to solve them is a good sign; enjoying solving them is an even better one.
Q:

What language should I learn first?
A:

XHTML (the latest dialect of HTML) if you don't already know it. There are a lot of glossy, hype-intensive bad HTML books out there, and distressingly few good ones. The one I like best is HTML: The Definitive Guide.

But HTML is not a full programming language. When you're ready to start programming, I would recommend starting with Python. You will hear a lot of people recommending Perl, and Perl is still more popular than Python, but it's harder to learn and (in my opinion) less well designed.

C is really important, but it's also much more difficult than either Python or Perl. Don't try to learn it first.

Windows users, do not settle for Visual Basic. It will teach you bad habits, and it's not portable off Windows. Avoid.
Q:

What kind of hardware do I need?
A:

It used to be that personal computers were rather underpowered and memory-poor, enough so that they placed artificial limits on a hacker's learning process. This stopped being true in the mid-1990s; any machine from an Intel 486DX50 up is more than powerful enough for development work, X, and Internet communications, and the smallest disks you can buy today are plenty big enough.

The important thing in choosing a machine on which to learn is whether its hardware is Linux-compatible (or BSD-compatible, should you choose to go that route). Again, this will be true for almost all modern machines. The only real sticky areas are modems and wireless cards; some machines have Windows-specific hardware that won't work with Linux.

There's a FAQ on hardware compatibility; the latest version is here.
Q:

I want to contribute. Can you help me pick a problem to work on?
A:

No, because I don't know your talents or interests. You have to be self-motivated or you won't stick, which is why having other people choose your direction almost never works.

Try this. Watch the project announcements scroll by on Freshmeat for a few days. When you see one that makes you think "Cool! I'd like to work on that!", join it.
Q:

Do I need to hate and bash Microsoft?
A:

No, you don't. Not that Microsoft isn't loathsome, but there was a hacker culture long before Microsoft and there will still be one long after Microsoft is history. Any energy you spend hating Microsoft would be better spent on loving your craft. Write good code — that will bash Microsoft quite sufficiently without polluting your karma.
Q:

But won't open-source software leave programmers unable to make a living?
A:

This seems unlikely — so far, the open-source software industry seems to be creating jobs rather than taking them away. If having a program written is a net economic gain over not having it written, a programmer will get paid whether or not the program is going to be open-source after it's done. And, no matter how much "free" software gets written, there always seems to be more demand for new and customized applications. I've written more about this at the Open Source pages.
Q:

Where can I get a free Unix?
A:

If you don't have a Unix installed on your machine yet, elsewhere on this page I include pointers to where to get the most commonly used free Unix. To be a hacker you need motivation and initiative and the ability to educate yourself. Start now...


How To Ask Questions The Smart Way
Eric Steven Raymond
Thyrsus Enterprises




Rick Moen





Copyright © 2001,2006 Eric S. Raymond, Rick Moen
Revision History
Revision 3.3 29 Sep 2006 esr
Folded in a good suggestion from Kai Niggemann.
Revision 3.2 10 Jan 2006 esr
Folded in edits from Rick Moen.
Revision 3.1 28 Oct 2004 esr
Document 'Google is your friend!'
Revision 3.0 2 Feb 2004 esr
Major addition of stuff about proper etiquette on Web forums.

Table of Contents

Translations
Disclaimer
Introduction
Before You Ask
When You Ask

Choose your forum carefully
Web and IRC forums directed towards newbies often give the quickest response
As a second step, use project mailing lists
Use meaningful, specific subject headers
Make it easy to reply
Write in clear, grammatical, correctly-spelled language
Send questions in accessible, standard formats
Be precise and informative about your problem
Volume is not precision
Don't claim that you have found a bug
Grovelling is not a substitute for doing your homework
Describe the problem's symptoms, not your guesses
Describe your problem's symptoms in chronological order
Describe the goal, not the step
Don't ask people to reply by private e-mail
Be explicit about your question
Don't post homework questions
Prune pointless queries
Don't flag your question as “Urgent”, even if it is for you
Courtesy never hurts, and sometimes helps
Follow up with a brief note on the solution

How To Interpret Answers

RTFM and STFW: How To Tell You've Seriously Screwed Up
If you don't understand...
Dealing with rudeness

On Not Reacting Like A Loser
Questions Not To Ask
Good and Bad Questions
If You Can't Get An Answer
How To Answer Questions in a Helpful Way
Related Resources
Acknowledgements

Translations

Translations: Bahasa Indonesian Brazilo-Portuguese Chinese Czech Danish Estonian Finnish French German Hebrew Hungarian Italian Japanese Polish Portuguese Romanian Russian Serbian Spanish Swedish Turkish. If you want to copy, mirror, translate, or excerpt this document, please see my copying policy.
Disclaimer

Many project websites link to this document in their sections on how to get help. That's fine, it's the use we intended — but if you are a webmaster creating such a link for your project page, please display prominently near the link notice that we are not a help desk for your project!

We have learned the hard way that without such a notice, we will repeatedly be pestered by idiots who think having published this document makes it our job to solve all the world's technical problems.

If you're reading this document because you need help, and you walk away with the impression you can get it directly from the authors, you are one of the idiots in question. Don't ask us questions. We'll just ignore you. We are here to show you how to get help from people who actually know about the software or hardware you're dealing with, but 99% of the time that will not be us. Unless you know for certain that one of the authors is an expert on what you're dealing with, leave us alone and everybody will be happier.
Introduction

In the world of hackers, the kind of answers you get to your technical questions depends as much on the way you ask the questions as on the difficulty of developing the answer. This guide will teach you how to ask questions in a way more likely to get you a satisfactory answer.

Now that use of open source has become widespread, you can often get as good answers from other, more experienced users as from hackers. This is a Good Thing; users tend to be just a little bit more tolerant of the kind of failures newbies often have. Still, treating experienced users like hackers in the ways we recommend here will generally be the most effective way to get useful answers out of them, too.

The first thing to understand is that hackers actually like hard problems and good, thought-provoking questions about them. If we didn't, we wouldn't be here. If you give us an interesting question to chew on we'll be grateful to you; good questions are a stimulus and a gift. Good questions help us develop our understanding, and often reveal problems we might not have noticed or thought about otherwise. Among hackers, “Good question!” is a strong and sincere compliment.

Despite this, hackers have a reputation for meeting simple questions with what looks like hostility or arrogance. It sometimes looks like we're reflexively rude to newbies and the ignorant. But this isn't really true.

What we are, unapologetically, is hostile to people who seem to be unwilling to think or to do their own homework before asking questions. People like that are time sinks — they take without giving back, and they waste time we could have spent on another question more interesting and another person more worthy of an answer. We call people like this “losers” (and for historical reasons we sometimes spell it “lusers”).

We realize that there are many people who just want to use the software we write, and who have no interest in learning technical details. For most people, a computer is merely a tool, a means to an end; they have more important things to do and lives to live. We acknowledge that, and don't expect everyone to take an interest in the technical matters that fascinate us. Nevertheless, our style of answering questions is tuned for people who do take such an interest and are willing to be active participants in problem-solving. That's not going to change. Nor should it; if it did, we would become less effective at the things we do best.

We're (largely) volunteers. We take time out of busy lives to answer questions, and at times we're overwhelmed with them. So we filter ruthlessly. In particular, we throw away questions from people who appear to be losers in order to spend our question-answering time more efficiently, on winners.

If you find this attitude obnoxious, condescending, or arrogant, check your assumptions. We're not asking you to genuflect to us — in fact, most of us would love nothing more than to deal with you as an equal and welcome you into our culture, if you put in the effort required to make that possible. But it's simply not efficient for us to try to help people who are not willing to help themselves. It's OK to be ignorant; it's not OK to play stupid.

So, while it isn't necessary to already be technically competent to get attention from us, it is necessary to demonstrate the kind of attitude that leads to competence — alert, thoughtful, observant, willing to be an active partner in developing a solution. If you can't live with this sort of discrimination, we suggest you pay somebody for a commercial support contract instead of asking hackers to personally donate help to you.

If you decide to come to us for help, you don't want to be one of the losers. You don't want to seem like one, either. The best way to get a rapid and responsive answer is to ask it like a person with smarts, confidence, and clues who just happens to need help on one particular problem.

(Improvements to this guide are welcome. You can mail suggestions to esr@thyrsus.com or respond-auto@linuxmafia.com. Note however that this document is not intended to be a general guide to netiquette, and we will generally reject suggestions that are not specifically related to eliciting useful answers in a technical forum.)
Before You Ask

Before asking a technical question by e-mail, or in a newsgroup, or on a website chat board, do the following:

1.

Try to find an answer by searching the archives of the forum you plan to post to.
2.

Try to find an answer by searching the Web.
3.

Try to find an answer by reading the manual.
4.

Try to find an answer by reading a FAQ.
5.

Try to find an answer by inspection or experimentation.
6.

Try to find an answer by asking a skilled friend.
7.

If you're a programmer, try to find an answer by reading the source code.

When you ask your question, display the fact that you have done these things first; this will help establish that you're not being a lazy sponge and wasting people's time. Better yet, display what you have learned from doing these things. We like answering questions for people who have demonstrated they can learn from the answers.

Use tactics like doing a Google search on the text of whatever error message you get (searching Google groups as well as Web pages). This might well take you straight to fix documentation or a mailing list thread answering your question. Even if it doesn't, saying “I googled on the following phrase but didn't get anything that looked promising” is a good thing to do in e-mail or news postings requesting help, if only because it records what searches won't help.

Take your time. Do not expect to be able to solve a complicated problem with a few seconds of Googling. Read and understand the FAQs, sit back, relax and give the problem some thought before approaching experts. Trust us, they will be able to tell from your questions how mnuch reading and thinking you did, and will be more willing to help if you come prepared. Don't instantly fire your whole arsenal of questions just because your first search turned up no answers (or too many).

Prepare your question. Think it through. Hasty-sounding questions get hasty answers, or none at all. The more you do to demonstrate that having put thought and effort into solving your problem before seeking help, the more likely you are to actually get help.

Beware of asking the wrong question. If you ask one that is based on faulty assumptions, J. Random Hacker is quite likely to reply with a uselessly literal answer while thinking “Stupid question...”, and hoping the experience of getting what you asked for rather than what you needed will teach you a lesson.

Never assume you are entitled to an answer. You are not; you aren't, after all, paying for the service. You will earn an answer, if you earn it, by asking a substantial, interesting, and thought-provoking question — one that implicitly contributes to the experience of the community rather than merely passively demanding knowledge from others.

On the other hand, making it clear that you are able and willing to help in the process of developing the solution is a very good start. “Would someone provide a pointer?”, “What is my example missing?”, and “What site should I have checked?” are more likely to get answered than “Please post the exact procedure I should use.” because you're making it clear that you're truly willing to complete the process if someone can just point you in the right direction.
When You Ask
Choose your forum carefully

Be sensitive in choosing where you ask your question. You are likely to be ignored, or written off as a loser, if you:

*

post your question to a forum where it's off topic
*

post a very elementary question to a forum where advanced technical questions are expected, or vice-versa
*

cross-post to too many different newsgroups
*

post a personal e-mail to somebody who is neither an acquaintance of yours nor personally responsible for solving your problem

Hackers blow off questions that are inappropriately targeted in order to try to protect their communications channels from being drowned in irrelevance. You don't want this to happen to you.

The first step, therefore, is to find the right forum. Again, Google and other Web-searching methods are your friend. Use them to find the project webpage most closely associated with the hardware or software giving you difficulties. Usually it will have links to a FAQ (Frequently Asked Questions) list, and to project mailing lists and their archives. These mailing lists are the final places to go for help, if your own efforts (including reading those FAQs you found) do not find you a solution. The project page may also describe a bug-reporting procedure, or have a link to one; if so, follow it.

Shooting off an e-mail to a person or forum which you are not familiar with is risky at best. For example, do not assume that the author of an informative webpage wants to be your free consultant. Do not make optimistic guesses about whether your question will be welcome — if you're unsure, send it elsewhere, or refrain from sending it at all.

When selecting a Web forum, newsgroup or mailing list, don't trust the name by itself too far; look for a FAQ or charter to verify your question is on-topic. Read some of the back traffic before posting so you'll get a feel for how things are done there. In fact, it's a very good idea to do a keyword search for words relating to your problem on the newsgroup or mailing list archives before you post. It may find you an answer, and if not it will help you formulate a better question.

Don't shotgun-blast all the available help channels at once, that's like yelling and irritates people. Step through them.

Know what your topic is! One of the classic mistakes is asking questions about the Unix or Windows programming interface in a forum devoted to a language or library or tool portable across both. If you don't understand why this is a blunder, you'd be best off not asking any questions at all until you get it.

In general, questions to a well-selected public forum are more likely to get useful answers than equivalent questions to a private one. There are multiple reasons for this. One is simply the size of the pool of potential respondents. Another is the size of the audience; hackers would rather answer questions that educate many people than questions serving only a few.

Understandably, skilled hackers and authors of popular software are already receiving more than their fair share of mis-targeted messages. By adding to the flood, you could in extreme cases even be the straw that breaks the camel's back — quite a few times, contributors to popular projects have withdrawn their support because collateral damage in the form of useless e-mail traffic to their personal accounts became unbearable.
Web and IRC forums directed towards newbies often give the quickest response

Your local user group, or your Linux distribution, may advertise a Web forum or IRC channel where newbies can get help. (In non-English-speaking countries newbie forums are still more likely to be mailing lists.) These are good first places to ask, especially if you think you may have tripped over a relatively simple or common problem. An advertised IRC channel is an open invitation to ask questions there and often get answers in real time.

In fact, if you got the program that is giving you problems from a Linux distribution (as common today), it may be better to ask in the distro's forum/list before trying the program's project forum/list. The project's hackers may just say, “use our build”.

Before posting to any Web forum, check if it has a Search feature. If it does, try a couple of keyword searches for something like your problem; it just might help. If you did a general Web search before (as you should have), search the forum anyway; your Web-wide search engine might not have all of this forum indexed recently.

There is an increasing tendency for projects to do user support over a Web forum or IRC channel, with e-mail reserved more for development traffic. So look for those channels first when seeking project-specific help.
As a second step, use project mailing lists

When a project has a development mailing list, write to the mailing list, not to individual developers, even if you believe you know who can best answer your question. Check the documentation of the project and its homepage for the address of a project mailing list, and use it. There are several good reasons for this policy:

*

Any question good enough to be asked of one developer will also be of value to the whole group. Contrariwise, if you suspect your question is too dumb for a mailing list, it's not an excuse to harass individual developers.
*

Asking questions on the list distributes load among developers. The individual developer (especially if he's the project leader) may be too busy to answer your questions.
*

Most mailing lists are archived and the archives are indexed by search engines. If you ask your question on-list and it is answered, a future querent could find your question and the answer on the Web instead of asking it again.
*

If certain questions are seen to be asked often, developers can use that information to improve the documentation or the software itself to be less confusing. But if those questions are asked in private, nobody has the complete picture of what questions are asked most often.

If a project has both a “user” and a “developer” (or “hacker”) mailing list or Web forum, and you are not hacking on the code, ask in the “user” list/forum. Do not assume that you will be welcome on the developer list, where they're likely to experience your question as noise disrupting their developer traffic.

However, if you are sure your question is non-trivial, and you get no answer in the “user” list/forum for several days, try the “developer” one. You would be well advised to lurk there for a few days before posting to learn the local folkways (actually this is good advice on any private or semi-private list).

If you cannot find a project's mailing list address, but only see the address of the maintainer of the project, go ahead and write to the maintainer. But even in that case, don't assume that the mailing list doesn't exist. Mention in your e-mail that you tried and could not find the appropriate mailing list. Also mention that you don't object to having your message forwarded to other people. (Many people believe that private e-mail should remain private, even if there is nothing secret in it. By allowing your message to be forwarded you give your correspondent a choice about how to handle your e-mail.)
Use meaningful, specific subject headers

On mailing lists, newsgroups or Web forums, the subject header is your golden opportunity to attract qualified experts' attention in around 50 characters or fewer. Don't waste it on babble like “Please help me” (let alone “PLEASE HELP ME!!!!”; messages with subjects like that get discarded by reflex). Don't try to impress us with the depth of your anguish; use the space for a super-concise problem description instead.

One good convention for subject headers, used by many tech support organizations, is “object - deviation”. The “object” part specifies what thing or group of things is having a problem, and the “deviation” part describes the deviation from expected behavior.

Stupid:

HELP! Video doesn't work properly on my laptop!
Smart:

X.org 6.8.1 misshapen mouse cursor, Fooware MV1005 vid. chipset
Smarter:

X.org 6.8.1 mouse cursor on Fooware MV1005 vid. chipset - is misshapen

The process of writing an “object-deviation” description will help you organize your thinking about the problem in more detail. What is affected? Just the mouse cursor or other graphics too? Is this specific to the X.org version of X? To version 6.8.1? Is this specific to Fooware video chipsets? To model MV1005? A hacker who sees the result can immediately understand what it is that you are having a problem with and the problem you are having, at a glance.

More generally, imagine looking at the index of an archive of questions, with just the subject lines showing. Make your subject line reflect your question well enough that the next guy searching the archive with a question similar to yours will be able to follow the thread to an answer rather than posting the question again.

If you ask a question in a reply, be sure to change the subject line to indicate that you're asking a question. A Subject line that looks like “Re: test” or “Re: new bug” is less likely to attract useful amounts of attention. Also, pare quotation of previous messages to the minimum consistent with cluing in new readers.

Do not simply hit reply to a list message in order to start an entirely new thread. This will limit your audience. Some mail readers, like mutt, allow the user to sort by thread and then hide messages in a thread by folding the thread. Folks who do that will never see your message.

Changing the subject is not sufficient. Mutt, and probably other mail readers, looks at other information in the e-mail's headers to assign it to a thread, not the subject line. Instead start an entirely new e-mail.

On Web forums the rules of good practice are slightly different, because messages are usually much more tightly bound to specific discussion threads and often invisible outside those threads. Changing the subject when asking a question in reply is not essential. Not all forums even allow separate subject lines on replies, and nearly nobody reads them when they do. However, asking a question in a reply is a dubious practice in itself, because it will only be seen by those who are watching this thread. So, unless you are sure you want to ask only the people currently active in the thread, start a new one.
Make it easy to reply

Finishing your query with “Please send your reply to... ” makes it quite unlikely you will get an answer. If you can't be bothered to take even the few seconds required to set up a correct Reply-To header in your mail agent, we can't be bothered to take even a few seconds to think about your problem. If your mail program doesn't permit this, get a better mail program. If your operating system doesn't support any e-mail programs that permit this, get a better operating system.

In Web forums, asking for a reply by e-mail is outright rude, unless you believe the information may be sensitive (and somebody will, for some unknown reason, let you but not the whole forum know it). If you want an e-mail copy when somebody replies in the thread, request that the Web forum send it; this feature is supported almost everywhere under options like “watch this thread”, “send e-mail on answers”, etc.
Write in clear, grammatical, correctly-spelled language

We've found by experience that people who are careless and sloppy writers are usually also careless and sloppy at thinking and coding (often enough to bet on, anyway). Answering questions for careless and sloppy thinkers is not rewarding; we'd rather spend our time elsewhere.

So expressing your question clearly and well is important. If you can't be bothered to do that, we can't be bothered to pay attention. Spend the extra effort to polish your language. It doesn't have to be stiff or formal — in fact, hacker culture values informal, slangy and humorous language used with precision. But it has to be precise; there has to be some indication that you're thinking and paying attention.

Spell, punctuate, and capitalize correctly. Don't confuse “its” with “it's”, “loose” with “lose”, or “discrete” with “discreet”. Don't TYPE IN ALL CAPS; this is read as shouting and considered rude. (All-smalls is only slightly less annoying, as it's difficult to read. Alan Cox can get away with it, but you can't.)

More generally, if you write like a semi-literate boob you will very likely be ignored. Writing like a l33t script kiddie hax0r is the absolute kiss of death and guarantees you will receive nothing but stony silence (or, at best, a heaping helping of scorn and sarcasm) in return.

If you are asking questions in a forum that does not use your native language, you will get a limited amount of slack for spelling and grammar errors — but no extra slack at all for laziness (and yes, we can usually spot that difference). Also, unless you know what your respondent's languages are, write in English. Busy hackers tend to simply flush questions in languages they don't understand, and English is the working language of the Internet. By writing in English you minimize your chances that your question will be discarded unread.
Send questions in accessible, standard formats

If you make your question artificially hard to read, it is more likely to be passed over in favor of one that isn't. So:

*

Send plain text mail, not HTML. (It's not hard to turn off HTML.)
*

MIME attachments are usually OK, but only if they are real content (such as an attached source file or patch), and not merely boilerplate generated by your mail client (such as another copy of your message).
*

Don't send e-mail in which entire paragraphs are single multiply-wrapped lines. (This makes it too difficult to reply to just part of the message.) Assume that your respondents will be reading mail on 80-character-wide text displays and set your line wrap accordingly, to something less than 80.
*

However, do not wrap data (such as log file dumps or session transcripts) at any fixed column width. Data should be included as-is, so respondents can have confidence that they are seeing what you saw.
*

Don't send MIME Quoted-Printable encoding to an English-language forum. This encoding can be necessary when you're posting in a language ASCII doesn't cover, but many e-mail agents don't support it. When they break, all those =20 glyphs scattered through the text are ugly and distracting — or may actively sabotage the semantics of your text.
*

Never, ever expect hackers to be able to read closed proprietary document formats like Microsoft Word or Excel. Most hackers react to these about as well as you would to having a pile of steaming pig manure dumped on your doorstep. Even when they can cope, they resent having to do so.
*

If you're sending e-mail from a Windows machine, turn off Microsoft's stupid “Smart Quotes” feature. This is so you'll avoid sprinkling garbage characters through your mail.
*

In Web forums, do not abuse “smiley” and “HTML” features (when they are present). A smiley or two is usually OK, but colored fancy text tends to make people think you are lame. Seriously overusing smileys and color and fonts will make you come off like a giggly teenage girl, which is not generally a good idea unless you are more interested in sex than answers.

If you're using a graphical-user-interface mail client such as Netscape Messenger, MS Outlook, or their ilk, beware that it may violate these rules when used with its default settings. Most such clients have a menu-based “View Source” command. Use this on something in your sent-mail folder, verifying sending of plain text without unnecessary attached crud.
Be precise and informative about your problem

*

Describe the symptoms of your problem or bug carefully and clearly.
*

Describe the environment in which it occurs (machine, OS, application, whatever). Provide your vendor's distribution and release level (e.g.: “Fedora Core 4”, “Slackware 9.1”, etc.).
*

Describe the research you did to try and understand the problem before you asked the question.
*

Describe the diagnostic steps you took to try and pin down the problem yourself before you asked the question.
*

Describe any possibly relevant recent changes in your computer or software configuration.

Do the best you can to anticipate the questions a hacker will ask, and answer them in advance in your request for help.

Simon Tatham has written an excellent essay entitled How to Report Bugs Effectively. I strongly recommend that you read it.
Volume is not precision

You need to be precise and informative. This end is not served by simply dumping huge volumes of code or data into a help request. If you have a large, complicated test case that is breaking a program, try to trim it and make it as small as possible.

This is useful for at least three reasons. One: being seen to invest effort in simplifying the question makes it more likely you'll get an answer, Two: simplifying the question makes it more likely you'll get a useful answer. Three: In the process of refining your bug report, you may develop a fix or workaround yourself.
Don't claim that you have found a bug

When you are having problems with a piece of software, don't claim you have found a bug unless you are very, very sure of your ground. Hint: unless you can provide a source-code patch that fixes the problem, or a regression test against a previous version that demonstrates incorrect behavior, you are probably not sure enough. This applies to webpages and documentation, too; if you have found a documentation “bug”, you should supply replacement text and which pages it should go on.

Remember, there are many other users that are not experiencing your problem. Otherwise you would have learned about it while reading the documentation and searching the Web (you did do that before complaining, didn't you?). This means that very probably it is you who are doing something wrong, not the software.

The people who wrote the software work very hard to make it work as well as possible. If you claim you have found a bug, you'll be impugning their competence, which may offend some of them even if you are correct. It's especially undiplomatic to yell “bug” in the Subject line.

When asking your question, it is best to write as though you assume you are doing something wrong, even if you are privately pretty sure you have found an actual bug. If there really is a bug, you will hear about it in the answer. Play it so the maintainers will want to apologize to you if the bug is real, rather than so that you will owe them an apology if you have messed up.
Grovelling is not a substitute for doing your homework

Some people who get that they shouldn't behave rudely or arrogantly, demanding an answer, retreat to the opposite extreme of grovelling. “I know I'm just a pathetic newbie loser, but...”. This is distracting and unhelpful. It's especially annoying when it's coupled with vagueness about the actual problem.

Don't waste your time, or ours, on crude primate politics. Instead, present the background facts and your question as clearly as you can. That is a better way to position yourself than by grovelling.

Sometimes Web forums have separate places for newbie questions. If you feel you do have a newbie question, just go there. But don't grovel there either.
Describe the problem's symptoms, not your guesses

It's not useful to tell hackers what you think is causing your problem. (If your diagnostic theories were such hot stuff, would you be consulting others for help?) So, make sure you're telling them the raw symptoms of what goes wrong, rather than your interpretations and theories. Let them do the interpretation and diagnosis. If you feel it's important to state your guess, clearly label it as such and describe why that answer isn't working for you.

Stupid:

I'm getting back-to-back SIG11 errors on kernel compiles, and suspect a hairline crack on one of the motherboard traces. What's the best way to check for those?
Smart:

My home-built K6/233 on an FIC-PA2007 motherboard (VIA Apollo VP2 chipset) with 256MB Corsair PC133 SDRAM starts getting frequent SIG11 errors about 20 minutes after power-on during the course of kernel compiles, but never in the first 20 minutes. Rebooting doesn't restart the clock, but powering down overnight does. Swapping out all RAM didn't help. The relevant part of a typical compile session log follows.

Since the preceding point seems to be a tough one for many people to grasp, here's a phrase to remind you: "All diagnosticians are from Missouri." That US state's official motto is "Show me" (earned in 1899, when Congressman Willard D. Vandiver said "I come from a country that raises corn and cotton and cockleburs and Democrats, and frothy eloquence neither convinces nor satisfies me. I'm from Missouri. You've got to show me.") In diagnosticians' case, it's not a matter of skepticism, but rather a literal, functional need to see whatever is as close as possible to the same raw evidence that you see, rather than your surmises and summaries. Show us.
Describe your problem's symptoms in chronological order

The clues most useful in figuring out something that went wrong often lie in the events immediately prior. So, your account should describe precisely what you did, and what the machine did, leading up to the blowup. In the case of command-line processes, having a session log (e.g., using the script utility) and quoting the relevant twenty or so lines is very useful.

If the program that blew up on you has diagnostic options (such as -v for verbose), try to select options that will add useful debugging information to the transcript. Remember that more is not necessarily better; try to choose a debug level that will inform rather than drowning the reader in junk.

If your account ends up being long (more than about four paragraphs), it might be useful to succinctly state the problem up top, then follow with the chronological tale. That way, hackers will know what to watch for in reading your account.
Describe the goal, not the step

If you are trying to find out how to do something (as opposed to reporting a bug), begin by describing the goal. Only then describe the particular step towards it that you are blocked on.

Often, people who need technical help have a high-level goal in mind and get stuck on what they think is one particular path towards the goal. They come for help with the step, but don't realize that the path is wrong. It can take substantial effort to get past this.

Stupid:

How do I get the color-picker on the FooDraw program to take a hexadecimal RGB value?
Smart:

I'm trying to replace the color table on an image with values of my choosing. Right now the only way I can see to do this is by editing each table slot, but I can't get FooDraw's color picker to take a hexadecimal RGB value.

The second version of the question is smart. It allows an answer that suggests a tool better suited to the task.
Don't ask people to reply by private e-mail

Hackers believe solving problems should be a public, transparent process during which a first try at an answer can and should be corrected if someone more knowledgeable notices that it is incomplete or incorrect. Also, helpers get some of their reward for being respondents from being seen to be competent and knowledgeable by their peers.

When you ask for a private reply, you are disrupting both the process and the reward. Don't do this. It's the respondent's choice whether to reply privately — and if he does, it's usually because he thinks the question is too ill-formed or obvious to be interesting to others.

There is one limited exception to this rule. If you think the question is such that you are likely to get many answers that are all closely similar, then the magic words are “e-mail me and I'll summarize the answers for the group”. It is courteous to try and save the mailing list or newsgroup a flood of substantially identical postings — but you have to keep the promise to summarize.
Be explicit about your question

Open-ended questions tend to be perceived as open-ended time sinks. Those people most likely to be able to give you a useful answer are also the busiest people (if only because they take on the most work themselves). People like that are allergic to open-ended time sinks, thus they tend to be allergic to open-ended questions.

You are more likely to get a useful response if you are explicit about what you want respondents to do (provide pointers, send code, check your patch, whatever). This will focus their effort and implicitly put an upper bound on the time and energy a respondent must allocate to helping you. This is good.

To understand the world the experts live in, think of expertise as an abundant resource and time to respond as a scarce one. The less of a time commitment you implicitly ask for, the more likely you are to get an answer from someone really good and really busy.

So it is useful to frame your question to minimize the time commitment required for an expert to field it — but this is often not the same thing as simplifying the question. Thus, for example, “Would you give me a pointer to a good explanation of X?” is usually a smarter question than “Would you explain X, please?”. If you have some malfunctioning code, it is usually smarter to ask for someone to explain what's wrong with it than it is to ask someone to fix it.
Don't post homework questions

Hackers are good at spotting homework questions; most of us have done them ourselves. Those questions are for you to work out, so that you will learn from the experience. It is OK to ask for hints, but not for entire solutions.

If you suspect you have been passed a homework question, but can't solve it anyway, try asking in a user group forum or (as a last resort) in a “user” list/forum of a project. While the hackers will spot it, some of the advanced users may at least give you a hint.
Prune pointless queries

Resist the temptation to close your request for help with semantically-null questions like “Can anyone help me?” or “Is there an answer?” First: if you've written your problem description halfway competently, such tacked-on questions are at best superfluous. Second: because they are superfluous, hackers find them annoying — and are likely to return logically impeccable but dismissive answers like “Yes, you can be helped” and “No, there is no help for you.”

In general, asking yes-or-no questions is a good thing to avoid unless you want a yes-or-no answer.
Don't flag your question as “Urgent”, even if it is for you

That's your problem, not ours. Claiming urgency is very likely to be counter-productive: most hackers will simply delete such messages as rude and selfish attempts to elicit immediate and special attention.

There is one semi-exception. It can be worth mentioning if you're using the program in some high-profile place, one that the hackers will get excited about; in such a case, if you're under time pressure, and you say so politely, people may get interested enough to answer faster.

This is a very risky thing to do, however, because the hackers' metric for what is exciting probably differs from yours. Posting from the International Space Station would qualify, for example, but posting on behalf of a feel-good charitable or political cause would almost certainly not. In fact, posting “Urgent: Help me save the fuzzy baby seals!” will reliably get you shunned or flamed even by hackers who think fuzzy baby seals are important.

If you find this mysterious, re-read the rest of this how-to repeatedly until you understand it before posting anything at all.
Courtesy never hurts, and sometimes helps

Be courteous. Use “Please” and “Thanks for your attention” or “Thanks for your consideration”. Make it clear you appreciate the time people spend helping you for free.

To be honest, this isn't as important as (and cannot substitute for) being grammatical, clear, precise and descriptive, avoiding proprietary formats etc.; hackers in general would rather get somewhat brusque but technically sharp bug reports than polite vagueness. (If this puzzles you, remember that we value a question by what it teaches us.)

However, if you've got your technical ducks in a row, politeness does increase your chances of getting a useful answer.

(We must note that the only serious objection we've received from veteran hackers to this HOWTO is with respect to our previous recommendation to use “Thanks in advance”. Some hackers feel this connotes an intention not to thank anybody afterwards. Our recommendation is to either say “Thanks in advance” first and thank respondents afterwards, or express courtesy in a different way, such as by saying “Thanks for your attention” or “Thanks for your consideration”.)
Follow up with a brief note on the solution

Send a note after the problem has been solved to all who helped you; let them know how it came out and thank them again for their help. If the problem attracted general interest in a mailing list or newsgroup, it's appropriate to post the followup there.

Optimally, the reply should be to the thread started by the original question posting, and should have ‘FIXED’, ‘RESOLVED’ or an equally obvious tag in the subject line. On mailing lists with fast turnaround, a potential respondent who sees a thread about “Problem X” ending with “Problem X - FIXED” knows not to waste his/her time even reading the thread (unless (s)he) personally finds Problem X interesting) and can therefore use that time solving a different problem.

Your followup doesn't have to be long and involved; a simple “Howdy — it was a failed network cable! Thanks, everyone. - Bill” would be better than nothing. In fact, a short and sweet summary is better than a long dissertation unless the solution has real technical depth. Say what action solved the problem, but you need not replay the whole troubleshooting sequence.

For problems with some depth, it is appropriate to post a summary of the troubleshooting history. Describe your final problem statement. Describe what worked as a solution, and indicate avoidable blind alleys after that. The blind alleys should come after the correct solution and other summary material, rather than turning the follow-up into a detective story. Name the names of people who helped you; you'll make friends that way.

Besides being courteous and informative, this sort of followup will help others searching the archive of the mailing-list/newsgroup/forum to know exactly which solution helped you and thus may also help them.

Last, and not least, this sort of followup helps everybody who assisted feel a satisfying sense of closure about the problem. If you are not a techie or hacker yourself, trust us that this feeling is very important to the gurus and experts you tapped for help. Problem narratives that trail off into unresolved nothingness are frustrating things; hackers itch to see them resolved. The goodwill that scratching that itch earns you will be very, very helpful to you next time you need to pose a question.

Consider how you might be able to prevent others from having the same problem in the future. Ask yourself if a documentation or FAQ patch would help, and if the answer is yes send that patch to the maintainer.

Among hackers, this sort of good followup behavior is actually more important than conventional politeness. It's how you get a reputation for playing well with others, which can be a very valuable asset.
How To Interpret Answers
RTFM and STFW: How To Tell You've Seriously Screwed Up

There is an ancient and hallowed tradition: if you get a reply that reads “RTFM”, the person who sent it thinks you should have Read The Fucking Manual. He or she is almost certainly right. Go read it.

RTFM has a younger relative. If you get a reply that reads “STFW”, the person who sent it thinks you should have Searched The Fucking Web. He or she is almost certainly right. Go search it. (The milder version of this is when you are told “Google is your friend!”)

In Web forums, you may also be told to search the forum archives. In fact, someone may even be so kind as to provide a pointer to the previous thread where this problem was solved. But do not rely on this consideration; do your archive-searching before asking.

Often, the person telling you to do a search has the manual or the web page with the information you need open, and is looking at it as he or she types. These replies mean that he thinks (a) the information you need is easy to find, and (b) you will learn more if you seek out the information than if you have it spoon-fed to you.

You shouldn't be offended by this; by hacker standards, your respondent is showing you a rough kind of respect simply by not ignoring you. You should instead be thankful for this grandmotherly kindness.
If you don't understand...

If you don't understand the answer, do not immediately bounce back a demand for clarification. Use the same tools that you used to try and answer your original question (manuals, FAQs, the Web, skilled friends) to understand the answer. Then, if you still need to ask for clarification, exhibit what you have learned.

For example, suppose I tell you: “It sounds like you've got a stuck zentry; you'll need to clear it.” Then: here's a bad followup question: “What's a zentry?” Here's a good followup question: “OK, I read the man page and zentries are only mentioned under the -z and -p switches. Neither of them says anything about clearing zentries. Is it one of these or am I missing something here?”
Dealing with rudeness

Much of what looks like rudeness in hacker circles is not intended to give offense. Rather, it's the product of the direct, cut-through-the-bullshit communications style that is natural to people who are more concerned about solving problems than making others feel warm and fuzzy.

When you perceive rudeness, try to react calmly. If someone is really acting out, it is very likely a senior person on the list or newsgroup or forum will call him or her on it. If that doesn't happen and you lose your temper, it is likely that the person you lose it at was behaving within the hacker community's norms and you will be considered at fault. This will hurt your chances of getting the information or help you want.

On the other hand, you will occasionally run across rudeness and posturing that is quite gratuitous. The flip-side of the above is that it is acceptable form to slam real offenders quite hard, dissecting their misbehavior with a sharp verbal scalpel. Be very, very sure of your ground before you try this, however. The line between correcting an incivility and starting a pointless flamewar is thin enough that hackers themselves not infrequently blunder across it; if you are a newbie or an outsider, your chances of avoiding such a blunder are low. If you're after information rather than entertainment, it's better to keep your fingers off the keyboard than to risk this.

(Some people assert that many hackers have a mild form of autism or Asperger's Syndrome, and are actually missing some of the brain circuitry that lubricates “normal” human social interaction. This may or may not be true. If you are not a hacker yourself, it may help you cope with our eccentricities if you think of us as being brain-damaged. Go right ahead. We won't care; we like being whatever it is we are, and generally have a healthy skepticism about clinical labels.)

In the next section, we'll talk about a different issue; the kind of “rudeness” you'll see when you misbehave.
On Not Reacting Like A Loser

Odds are you'll screw up a few times on hacker community forums — in ways detailed in this article, or similar. And you'll be told exactly how you screwed up, possibly with colourful asides. In public.

When this happens, the worst thing you can do is whine about the experience, claim to have been verbally assaulted, demand apologies, scream, hold your breath, threaten lawsuits, complain to people's employers, leave the toilet seat up, etc. Instead, here's what you do:

Get over it. It's normal. In fact, it's healthy and appropriate.

Community standards do not maintain themselves: They're maintained by people actively applying them, visibly, in public. Don't whine that all criticism should have been conveyed via private e-mail: That's not how it works. Nor is it useful to insist you've been personally insulted when someone comments that one of your claims was wrong, or that his views differ. Those are loser attitudes.

There have been hacker forums where, out of some misguided sense of hyper-courtesy, participants are banned from posting any fault-finding with another's posts, and told “Don't say anything if you're unwilling to help the user.” The resulting departure of clueful participants to elsewhere causes them to descend into meaningless babble and become useless as technical forums.

Exaggeratedly “friendly” (in that fashion) or useful: Pick one.

Remember: When that hacker tells you that you've screwed up, and (no matter how gruffly) tells you not to do it again, he's acting out of concern for (1) you and (2) his community. It would be much easier for him to ignore you and filter you out of his life. If you can't manage to be grateful, at least have a little dignity, don't whine, and don't expect to be treated like a fragile doll just because you're a newcomer with a theatrically hypersensitive soul and delusions of entitlement.

Sometimes people will attack you personally, flame without an apparent reason, etc., even if you don't screw up (or have only screwed up in their imagination). In this case, complaining is the way to really screw up.

These flamers are either lamers who don't have a clue but believe themselves to be experts, or would-be psychologists testing whether you'll screw up. The other readers either ignore them, or find ways to deal with them on their own. The flamers' behavior creates problems for themselves, which don't have to concern you.

Don't let yourself be drawn into a flamewar, either. Most flames are best ignored — after you've checked whether they are really flames, not pointers to the ways in which you have screwed up, and not cleverly ciphered answers to your real question (this happens as well).
Questions Not To Ask

Here are some classic stupid questions, and what hackers are thinking when they don't answer them.

Q: Where can I find program or resource X?
Q: How can I use X to do Y?
Q: How can I configure my shell prompt?
Q: Can I convert an AcmeCorp document into a TeX file using the Bass-o-matic file converter?
Q: My {program, configuration, SQL statement} doesn't work
Q: I'm having problems with my Windows machine. Can you help?
Q: My program doesn't work. I think system facility X is broken.
Q: I'm having problems installing Linux or X. Can you help?
Q: How can I crack root/steal channel-ops privileges/read someone's e-mail?

Q:

Where can I find program or resource X?
A:

The same place I'd find it, fool — at the other end of a web search. Ghod, doesn't everybody know how to use Google yet?
Q:

How can I use X to do Y?
A:

If what you want is to do Y, you should ask that question without pre-supposing the use of a method that may not be appropriate. Questions of this form often indicate a person who is not merely ignorant about X, but confused about what problem Y they are solving and too fixated on the details of their particular situation. It is generally best to ignore such people until they define their problem better.
Q:

How can I configure my shell prompt?
A:

If you're smart enough to ask this question, you're smart enough to RTFM and find out yourself.
Q:

Can I convert an AcmeCorp document into a TeX file using the Bass-o-matic file converter?
A:

Try it and see. If you did that, you'd (a) learn the answer, and (b) stop wasting my time.
Q:

My {program, configuration, SQL statement} doesn't work
A:

This is not a question, and I'm not interested in playing Twenty Questions to pry your actual question out of you — I have better things to do. On seeing something like this, my reaction is normally of one of the following:

*

do you have anything else to add to that?
*

oh, that's too bad, I hope you get it fixed.
*

and this has exactly what to do with me?

Q:

I'm having problems with my Windows machine. Can you help?
A:

Yes. Throw out that Microsoft trash and install an open-source operating system like Linux or BSD.

Note: you can ask questions related to Windows machines if they are about a program that does have an official Windows build, or interacts with Windows machines (i.e., Samba). Just don't be surprised by the reply that the problem is with Windows and not the program, because Windows is so broken in general that this is very often the case.
Q:

My program doesn't work. I think system facility X is broken.
A:

While it is possible that you are the first person to notice an obvious deficiency in system calls and libraries heavily used by hundreds or thousands of people, it is rather more likely that you are utterly clueless. Extraordinary claims require extraordinary evidence; when you make a claim like this one, you must back it up with clear and exhaustive documentation of the failure case.
Q:

I'm having problems installing Linux or X. Can you help?
A:

No. I'd need hands-on access to your machine to troubleshoot this. Go ask your local Linux user group for hands-on help. (You can find a list of user groups here.)

Note: questions about installing Linux may be appropriate if you're on a forum or mailing list about a particular distribution, and the problem is with that distro; or on local user groups forums. In this case, be sure to describe the exact details of the failure. But do careful searching first, with "linux" and all suspicious pieces of hardware.
Q:

How can I crack root/steal channel-ops privileges/read someone's e-mail?
A:

You're a lowlife for wanting to do such things and a moron for asking a hacker to help you.
Good and Bad Questions

Finally, I'm going to illustrate how to ask questions in a smart way by example; pairs of questions about the same problem, one asked in a stupid way and one in a smart way.

Stupid: Where can I find out stuff about the Foonly Flurbamatic?

This question just begs for "STFW" as a reply.
Smart: I used Google to try to find “Foonly Flurbamatic 2600” on the Web, but I got no useful hits. Can I get a pointer to programming information on this device?

This one has already STFWed, and sounds like he might have a real problem.

Stupid: I can't get the code from project foo to compile. Why is it broken?

The querent assumes that somebody else screwed up. Arrogant git...
Smart: The code from project foo doesn't compile under Nulix version 6.2. I've read the FAQ, but it doesn't have anything in it about Nulix-related problems. Here's a transcript of my compilation attempt; is it something I did?

The querent has specified the environment, read the FAQ, is showing the error, and is not assuming his problems are someone else's fault. This one might be worth some attention.

Stupid: I'm having problems with my motherboard. Can anybody help?

J. Random Hacker's response to this is likely to be “Right. Do you need burping and diapering, too?” followed by a punch of the delete key.
Smart: I tried X, Y, and Z on the S2464 motherboard. When that didn't work, I tried A, B, and C. Note the curious symptom when I tried C. Obviously the florbish is grommicking, but the results aren't what one might expect. What are the usual causes of grommicking on Athlon MP motherboards? Anybody got ideas for more tests I can run to pin down the problem?

This person, on the other hand, seems worthy of an answer. He/she has exhibited problem-solving intelligence rather than passively waiting for an answer to drop from on high.

In the last question, notice the subtle but important difference between demanding “Give me an answer” and “Please help me figure out what additional diagnostics I can run to achieve enlightenment.”

In fact, the form of that last question is closely based on a real incident that happened in August 2001 on the linux-kernel mailing list (lkml). I (Eric) was the one asking the question that time. I was seeing mysterious lockups on a Tyan S2462 motherboard. The list members supplied the critical information I needed to solve them.

By asking the question in the way I did, I gave people something to chew on; I made it easy and attractive for them to get involved. I demonstrated respect for my peers' ability and invited them to consult with me as a peer. I also demonstrated respect for the value of their time by telling them the blind alleys I had already run down.

Afterwards, when I thanked everyone and remarked how well the process had worked, an lkml member observed that he thought it had worked not because I'm a “name” on that list, but because I asked the question in the proper form.

Hackers are in some ways a very ruthless meritocracy; I'm certain he was right, and that if I had behaved like a sponge I would have been flamed or ignored no matter who I was. His suggestion that I write up the whole incident as instruction to others led directly to the composition of this guide.
If You Can't Get An Answer

If you can't get an answer, please don't take it personally that we don't feel we can help you. Sometimes the members of the asked group may simply not know the answer. No response is not the same as being ignored, though admittedly it's hard to spot the difference from outside.

In general, simply re-posting your question is a bad idea. This will be seen as pointlessly annoying. Have patience: the person with your answer may be in a different time-zone and asleep. Or it may be that your question wasn't well-formed to begin with.

There are other sources of help you can go to, often sources better adapted to a novice's needs.

There are many online and local user groups who are enthusiasts about the software, even though they may never have written any software themselves. These groups often form so that people can help each other and help new users.

There are also plenty of commercial companies you can contract with for help, both large and small (Red Hat and SpikeSource are two of the best known; there are many others). Don't be dismayed at the idea of having to pay for a bit of help! After all, if your car engine blows a head gasket, chances are you would take it to a repair shop and pay to get it fixed. Even if the software didn't cost you anything, you can't expect that support to always come for free.

For popular software like Linux, there are at least 10,000 users per developer. It's just not possible for one person to handle the support calls from over 10,000 users. Remember that even if you have to pay for support, you are still paying much less than if you had to buy the software as well (and support for closed-source software is usually more expensive and less competent than support for open-source software).
How To Answer Questions in a Helpful Way

Be gentle. Problem-related stress can make people seem rude or stupid even when they're not.

Reply to a first offender off-line. There is no need of public humiliation for someone who may have made an honest mistake. A real newbie may not know how to search archives or where the FAQ is stored or posted.

If you don't know for sure, say so! A wrong but authoritative-sounding answer is worse than none at all. Don't point anyone down a wrong path simply because it's fun to sound like an expert. Be humble and honest; set a good example for both the querent and your peers.

If you can't help, don't hinder. Don't make jokes about procedures that could trash the user's setup — the poor sap might interpret these as instructions.

Ask probing questions to elicit more details. If you're good at this, the querent will learn something — and so might you. Try to turn the bad question into a good one; remember we were all newbies once.

While just muttering RTFM is sometimes justified when replying to someone who is just a lazy slob, a pointer to documentation (even if it's just a suggestion to google for a key phrase) is better.

If you're going to answer the question at all, give good value. Don't suggest kludgy workarounds when somebody is using the wrong tool or approach. Suggest good tools. Reframe the question.

Help your community learn from the question. When you field a good question, ask yourself “How would the relevant documentation or FAQ have to change so that nobody has to answer this again?” Then send a patch to the document maintainer.

If you did research to answer the question, demonstrate your skills rather than writing as though you pulled the answer out of your butt. Answering one good question is like feeding a hungry person one meal, but teaching them research skills by example is teaching them to grow food for a lifetime.
Related Resources

If you need instruction in the basics of how personal computers, Unix, and the Internet work, see The Unix and Internet Fundamentals HOWTO.

When you release software or write patches for software, try to follow the guidelines in the Software Release Practice HOWTO.
Acknowledgements

Evelyn Mitchell contributed some example stupid questions and inspired the “How To Give A Good Answer” section. Mikhail Ramendik contributed some particularly valuable suggestions for improvements.

The Unix and Internet Fundamentals HOWTO
Eric Raymond




Revision History
Revision 2.9 2004-03-03 Revised by: esr
Minor updates.
Revision 1.0 1998-10-29 Revised by: esr
Initial revision.

This document describes the working basics of PC-class computers, Unix-like operating systems, and the Internet in non-technical language.

Table of Contents
1. Introduction

1.1. Purpose of this document
1.2. New versions of this document
1.3. Feedback and corrections
1.4. Related resources

2. Basic anatomy of your computer
3. What happens when you switch on a computer?
4. What happens when you log in?
5. What happens when you run programs from the shell?
6. How do input devices and interrupts work?
7. How does my computer do several things at once?
8. How does my computer keep processes from stepping on each other?

8.1. Virtual memory: the simple version
8.2. Virtual memory: the detailed version
8.3. The Memory Management Unit

9. How does my computer store things in memory?

9.1. Numbers
9.2. Characters

10. How does my computer store things on disk?

10.1. Low-level disk and file system structure
10.2. File names and directories
10.3. Mount points
10.4. How a file gets looked up
10.5. File ownership, permissions and security
10.6. How things can go wrong

11. How do computer languages work?

11.1. Compiled languages
11.2. Interpreted languages
11.3. P-code languages

12. How does the Internet work?

12.1. Names and locations
12.2. The Domain Name System
12.3. Packets and routers
12.4. TCP and IP
12.5. HTTP, an application protocol

13. To Learn More

Next
Introduction


1. Introduction
1.1. Purpose of this document

This document is intended to help Linux and Internet users who are learning by doing. While this is a great way to acquire specific skills, sometimes it leaves peculiar gaps in one's knowledge of the basics — gaps which can make it hard to think creatively or troubleshoot effectively, from lack of a good mental model of what is really going on.

I'll try to describe in clear, simple language how it all works. The presentation will be tuned for people using Unix or Linux on PC-class hardware. Nevertheless, I'll usually refer simply to ‘Unix’ here, as most of what I will describe is constant across platforms and across Unix variants.

I'm going to assume you're using an Intel PC. The details differ slightly if you're running an Alpha or PowerPC or some other Unix box, but the basic concepts are the same.

I won't repeat things, so you'll have to pay attention, but that also means you'll learn from every word you read. It's a good idea to just skim when you first read this; you should come back and reread it a few times after you've digested what you have learned.

This is an evolving document. I intend to keep adding sections in response to user feedback, so you should come back and review it periodically.
1.2. New versions of this document

New versions of the Unix and Internet Fundamentals HOWTO will be periodically posted to comp.os.linux.help and comp.os.linux.announce and news.answers. They will also be uploaded to various Linux WWW and FTP sites, including the LDP home page.

You can view the latest version of this on the World Wide Web via the URL http:http://www.tldp.org/HOWTO/Unix-and-Internet-Fundamentals-HOWTO/index.html.

This document has been translated into Polish.
1.3. Feedback and corrections

If you have questions or comments about this document, please feel free to mail Eric S. Raymond, at esr@thyrsus.com. I welcome any suggestions or criticisms. I especially welcome hyperlinks to more detailed explanations of individual concepts. If you find a mistake with this document, please let me know so I can correct it in the next version. Thanks.
1.4. Related resources

If you're reading this in order to learn how to hack, you should also read the How To Become A Hacker FAQ. It has links to some other useful resources.

2. Basic anatomy of your computer

Your computer has a processor chip inside it that does the actual computing. It has internal memory (what DOS/Windows people call "RAM" and Unix people often call "core"; the Unix term is a folk memory from when RAM consisted of ferrite-core donuts). The processor and memory live on the motherboard, which is the heart of your computer.

Your computer has a screen and keyboard. It has hard drives and a CD-ROM and maybe a floppy disk. Some of these devices are run by controller cards that plug into the motherboard and help the computer drive them; others are run by specialized chipsets directly on the motherboard that fulfill the same function as a controller card. Your keyboard is too simple to need a separate card; the controller is built into the keyboard chassis itself.

We'll go into some of the details of how these devices work later. For now, here are a few basic things to keep in mind about how they work together:

All the parts of your computer inside the case are connected by a bus. Physically, the bus is what you plug your controller cards into (the video card, the disk controller, a sound card if you have one). The bus is the data highway between your processor, your screen, your disk, and everything else.

(If you've seen references to ‘ISA’, ‘PCI’, and ‘PCMCIA’ in connection with PCs and have not understood them, these are bus types. ISA is, except in minor details, the same bus that was used on IBM's original PCs in 1980; it is passing out of use now. PCI, for Peripheral Component Interconnection, is the bus used on most modern PCs, and on modern Macintoshes as well. PCMCIA is a variant of ISA with smaller physical connectors used on laptop computers.)

The processor, which makes everything else go, can't actually see any of the other pieces directly; it has to talk to them over the bus. The only other subsystem that it has really fast, immediate access to is memory (the core). In order for programs to run, then, they have to be in core (in memory).

When your computer reads a program or data off the disk, what actually happens is that the processor uses the bus to send a disk read request to your disk controller. Some time later the disk controller uses the bus to signal the processor that it has read the data and put it in a certain location in memory. The processor can then use the bus to look at that data.

Your keyboard and screen also communicate with the processor via the bus, but in simpler ways. We'll discuss those later on. For now, you know enough to understand what happens when you turn on your computer.


3. What happens when you switch on a computer?

A computer without a program running is just an inert hunk of electronics. The first thing a computer has to do when it is turned on is start up a special program called an operating system. The operating system's job is to help other computer programs to work by handling the messy details of controlling the computer's hardware.

The process of bringing up the operating system is called booting (originally this was bootstrapping and alluded to the process of pulling yourself up "by your bootstraps"). Your computer knows how to boot because instructions for booting are built into one of its chips, the BIOS (or Basic Input/Output System) chip.

The BIOS chip tells it to look in a fixed place, usually on the lowest-numbered hard disk (the boot disk) for a special program called a boot loader (under Linux the boot loader is called Grub or LILO). The boot loader is pulled into memory and started. The boot loader's job is to start the real operating system.

The loader does this by looking for a kernel, loading it into memory, and starting it. When you boot Linux and see "LILO" on the screen followed by a bunch of dots, it is loading the kernel. (Each dot means it has loaded another disk block of kernel code.)

(You may wonder why the BIOS doesn't load the kernel directly — why the two-step process with the boot loader? Well, the BIOS isn't very smart. In fact it's very stupid, and Linux doesn't use it at all after boot time. It was originally written for primitive 8-bit PCs with tiny disks, and literally can't access enough of the disk to load the kernel directly. The boot loader step also lets you start one of several operating systems off different places on your disk, in the unlikely event that Unix isn't good enough for you.)

Once the kernel starts, it has to look around, find the rest of the hardware, and get ready to run programs. It does this by poking not at ordinary memory locations but rather at I/O ports — special bus addresses that are likely to have device controller cards listening at them for commands. The kernel doesn't poke at random; it has a lot of built-in knowledge about what it's likely to find where, and how controllers will respond if they're present. This process is called autoprobing.

Most of the messages you see at boot time are the kernel autoprobing your hardware through the I/O ports, figuring out what it has available to it and adapting itself to your machine. The Linux kernel is extremely good at this, better than most other Unixes and much better than DOS or Windows. In fact, many Linux old-timers think the cleverness of Linux's boot-time probes (which made it relatively easy to install) was a major reason it broke out of the pack of free-Unix experiments to attract a critical mass of users.

But getting the kernel fully loaded and running isn't the end of the boot process; it's just the first stage (sometimes called run level 1). After this first stage, the kernel hands control to a special process called ‘init’ which spawns several housekeeping processes.

The init process's first job is usually to check to make sure your disks are OK. Disk file systems are fragile things; if they've been damaged by a hardware failure or a sudden power outage, there are good reasons to take recovery steps before your Unix is all the way up. We'll go into some of this later on when we talk about how file systems can go wrong.

Init's next step is to start several daemons. A daemon is a program like a print spooler, a mail listener or a WWW server that lurks in the background, waiting for things to do. These special programs often have to coordinate several requests that could conflict. They are daemons because it's often easier to write one program that runs constantly and knows about all requests than it would be to try to make sure that a flock of copies (each processing one request and all running at the same time) don't step on each other. The particular collection of daemons your system starts may vary, but will almost always include a print spooler (a gatekeeper daemon for your printer).

The next step is to prepare for users. Init starts a copy of a program called getty to watch your console (and maybe more copies to watch dial-in serial ports). This program is what issues the login prompt to your console. Once all daemons and getty processes for each terminal are started, we're at run level 2. At this level, you can log in and run programs.

But we're not done yet. The next step is to start up various daemons that support networking and other services. Once that's done, we're at run level 3 and the system is fully ready for use.



4. What happens when you log in?

When you log in (give a name to getty) you identify yourself to the computer. It then runs a program called (naturally enough) login, which takes your password and checks to see if you are authorized to be using the machine. If you aren't, your login attempt will be rejected. If you are, login does a few housekeeping things and then starts up a command interpreter, the shell. (Yes, getty and login could be one program. They're separate for historical reasons not worth going into here.)

Here's a bit more about what the system does before giving you a shell (you'll need to know this later when we talk about file permissions). You identify yourself with a login name and password. That login name is looked up in a file called /etc/passwd, which is a sequence of lines each describing a user account.

One of these fields is an encrypted version of the account password (sometimes the encrypted fields are actually kept in a second /etc/shadow file with tighter permissions; this makes password cracking harder). What you enter as an account password is encrypted in exactly the same way, and the login program checks to see if they match. The security of this method depends on the fact that, while it's easy to go from your clear password to the encrypted version, the reverse is very hard. Thus, even if someone can see the encrypted version of your password, they can't use your account. (It also means that if you forget your password, there's no way to recover it, only to change it to something else you choose.)

Once you have successfully logged in, you get all the privileges associated with the individual account you are using. You may also be recognized as part of a group. A group is a named collection of users set up by the system administrator. Groups can have privileges independently of their members’ privileges. A user can be a member of multiple groups. (For details about how Unix privileges work, see the section below on permissions.)

(Note that although you will normally refer to users and groups by name, they are actually stored internally as numeric IDs. The password file maps your account name to a user ID; the /etc/group file maps group names to numeric group IDs. Commands that deal with accounts and groups do the translation automatically.)

Your account entry also contains your home directory, the place in the Unix file system where your personal files will live. Finally, your account entry also sets your shell, the command interpreter that login will start up to accept your commmands.


5. What happens when you run programs from the shell?

The shell is Unix's interpreter for the commands you type in; it's called a shell because it wraps around and hides the operating system kernel. It's an important feature of Unix that the shell and kernel are separate programs communicating through a small set of system calls. This makes it possible for there to be multiple shells, suiting different tastes in interfaces.

The normal shell gives you the ‘$’ prompt that you see after logging in (unless you've customized it to be something else). We won't talk about shell syntax and the easy things you can see on the screen here; instead we'll take a look behind the scenes at what's happening from the computer's point of view.

After boot time and before you run a program, you can think of your computer as containing a zoo of processes that are all waiting for something to do. They're all waiting on events. An event can be you pressing a key or moving a mouse. Or, if your machine is hooked to a network, an event can be a data packet coming in over that network.

The kernel is one of these processes. It's a special one, because it controls when the other user processes can run, and it is normally the only process with direct access to the machine's hardware. In fact, user processes have to make requests to the kernel when they want to get keyboard input, write to your screen, read from or write to disk, or do just about anything other than crunching bits in memory. These requests are known as system calls.

Normally all I/O goes through the kernel so it can schedule the operations and prevent processes from stepping on each other. A few special user processes are allowed to slide around the kernel, usually by being given direct access to I/O ports. X servers (the programs that handle other programs’ requests to do screen graphics on most Unix boxes) are the most common example of this. But we haven't gotten to an X server yet; you're looking at a shell prompt on a character console.

The shell is just a user process, and not a particularly special one. It waits on your keystrokes, listening (through the kernel) to the keyboard I/O port. As the kernel sees them, it echoes them to your screen. When the kernel sees an ‘Enter’ it passes your line of text to the shell. The shell tries to interpret those keystrokes as commands.

Let's say you type ‘ls’ and Enter to invoke the Unix directory lister. The shell applies its built-in rules to figure out that you want to run the executable command in the file /bin/ls. It makes a system call asking the kernel to start /bin/ls as a new child process and give it access to the screen and keyboard through the kernel. Then the shell goes to sleep, waiting for ls to finish.

When /bin/ls is done, it tells the kernel it's finished by issuing an exit system call. The kernel then wakes up the shell and tells it it can continue running. The shell issues another prompt and waits for another line of input.

Other things may be going on while your ‘ls’ is executing, however (we'll have to suppose that you're listing a very long directory). You might switch to another virtual console, log in there, and start a game of Quake, for example. Or, suppose you're hooked up to the Internet. Your machine might be sending or receiving mail while /bin/ls runs.


6. How do input devices and interrupts work?

Your keyboard is a very simple input device; simple because it generates small amounts of data very slowly (by a computer's standards). When you press or release a key, that event is signalled up the keyboard cable to raise a hardware interrupt.

It's the operating system's job to watch for such interrupts. For each possible kind of interrupt, there will be an interrupt handler, a part of the operating system that stashes away any data associated with them (like your keypress/keyrelease value) until it can be processed.

What the interrupt handler for your keyboard actually does is post the key value into a system area near the bottom of memory. There, it will be available for inspection when the operating system passes control to whichever program is currently supposed to be reading from the keyboard.

More complex input devices like disk or network cards work in a similar way. Earlier, I referred to a disk controller using the bus to signal that a disk request has been fulfilled. What actually happens is that the disk raises an interrupt. The disk interrupt handler then copies the retrieved data into memory, for later use by the program that made the request.

Every kind of interrupt has an associated priority level. Lower-priority interrupts (like keyboard events) have to wait on higher-priority interrupts (like clock ticks or disk events). Unix is designed to give high priority to the kinds of events that need to be processed rapidly in order to keep the machine's response smooth.

In your operating system's boot-time messages, you may see references to IRQ numbers. You may be aware that one of the common ways to misconfigure hardware is to have two different devices try to use the same IRQ, without understanding exactly why.

Here's the answer. IRQ is short for "Interrupt Request". The operating system needs to know at startup time which numbered interrupts each hardware device will use, so it can associate the proper handlers with each one. If two different devices try use the same IRQ, interrupts will sometimes get dispatched to the wrong handler. This will usually at least lock up the device, and can sometimes confuse the OS badly enough that it will flake out or crash.


7. How does my computer do several things at once?

It doesn't, actually. Computers can only do one task (or process) at a time. But a computer can change tasks very rapidly, and fool slow human beings into thinking it's doing several things at once. This is called timesharing.

One of the kernel's jobs is to manage timesharing. It has a part called the scheduler which keeps information inside itself about all the other (non-kernel) processes in your zoo. Every 1/60th of a second, a timer goes off in the kernel, generating a clock interrupt. The scheduler stops whatever process is currently running, suspends it in place, and hands control to another process.

1/60th of a second may not sound like a lot of time. But on today's microprocessors it's enough to run tens of thousands of machine instructions, which can do a great deal of work. So even if you have many processes, each one can accomplish quite a bit in each of its timeslices.

In practice, a program may not get its entire timeslice. If an interrupt comes in from an I/O device, the kernel effectively stops the current task, runs the interrupt handler, and then returns to the current task. A storm of high-priority interrupts can squeeze out normal processing; this misbehavior is called thrashing and is fortunately very hard to induce under modern Unixes.

In fact, the speed of programs is only very seldom limited by the amount of machine time they can get (there are a few exceptions to this rule, such as sound or 3-D graphics generation). Much more often, delays are caused when the program has to wait on data from a disk drive or network connection.

An operating system that can routinely support many simultaneous processes is called "multitasking". The Unix family of operating systems was designed from the ground up for multitasking and is very good at it — much more effective than Windows or the old Mac OS, which had multitasking bolted into them as an afterthought and do it rather poorly. Efficient, reliable multitasking is a large part of what makes Linux superior for networking, communications, and Web service.

8. How does my computer keep processes from stepping on each other?

The kernel's scheduler takes care of dividing processes in time. Your operating system also has to divide them in space, so that processes can't step on each others'; working memory. Even if you assume that all programs are trying to be cooperative, you don't want a bug in one of them to be able to corrupt others. The things your operating system does to solve this problem are called memory management.

Each process in your zoo needs its own area of memory, as a place to run its code from and keep variables and results in. You can think of this set as consisting of a read-only code segment (containing the process's instructions) and a writeable data segment (containing all the process's variable storage). The data segment is truly unique to each process, but if two processes are running the same code Unix automatically arranges for them to share a single code segment as an efficiency measure.
8.1. Virtual memory: the simple version

Efficiency is important, because memory is expensive. Sometimes you don't have enough to hold the entirety of all the programs the machine is running, especially if you are using a large program like an X server. To get around this, Unix uses a technique called virtual memory. It doesn't try to hold all the code and data for a process in memory. Instead, it keeps around only a relatively small working set; the rest of the process's state is left in a special swap space area on your hard disk.

Note that in the past, that "Sometimes" last paragraph ago was "Almost always" — the size of memory was typically small relative to the size of running programs, so swapping was frequent. Memory is far less expensive nowadays and even low-end machines have quite a lot of it. On modern single-user machines with 64MB of memory and up, it's possible to run X and a typical mix of jobs without ever swapping after they're initially loaded into core.
8.2. Virtual memory: the detailed version

Actually, the last section oversimplified things a bit. Yes, programs see most of your memory as one big flat bank of addresses bigger than physical memory, and disk swapping is used to maintain that illusion. But your hardware actually has no fewer than five different kinds of memory in it, and the differences between them can matter a good deal when programs have to be tuned for maximum speed. To really understand what goes on in your machine, you should learn how all of them work.

The five kinds of memory are these: processor registers, internal (or on-chip) cache, external (or off-chip) cache, main memory, and disk. And the reason there are so many kinds is simple: speed costs money. I have listed these kinds of memory in decreasing order of access time and increasing order of cost. Register memory is the fastest and most expensive and can be random-accessed about a billion times a second, while disk is the slowest and cheapest and can do about 100 random accesses a second.

Here's a full list reflecting early-2000 speeds for a typical desktop machine. While speed and capacity will go up and prices will drop, you can expect these ratios to remain fairly constant — and it's those ratios that shape the memory hierarchy.

Disk

Size: 13000MB Accesses: 100KB/sec
Main memory

Size: 256MB Accesses: 100M/sec
External cache

Size: 512KB Accesses: 250M/sec
Internal Cache

Size: 32KB Accesses: 500M/sec
Processor

Size: 28 bytes Accesses: 1000M/sec

We can't build everything out of the fastest kinds of memory. It would be way too expensive — and even if it weren't, fast memory is volatile. That is, it loses its marbles when the power goes off. Thus, computers have to have hard disks or other kinds of non-volatile storage that retains data when the power goes off. And there's a huge mismatch between the speed of processors and the speed of disks. The middle three levels of the memory hierarchy (internal cache, external cache, and main memory) basically exist to bridge that gap.

Linux and other Unixes have a feature called virtual memory. What this means is that the operating system behaves as though it has much more main memory than it actually does. Your actual physical main memory behaves like a set of windows or caches on a much larger "virtual" memory space, most of which at any given time is actually stored on disk in a special zone called the swap area. Out of sight of user programs, the OS is moving blocks of data (called "pages") between memory and disk to maintain this illusion. The end result is that your virtual memory is much larger but not too much slower than real memory.

How much slower virtual memory is than physical depends on how well the operating system's swapping algorithms match the way your programs use virtual memory. Fortunately, memory reads and writes that are close together in time also tend to cluster in memory space. This tendency is called locality, or more formally locality of reference — and it's a good thing. If memory references jumped around virtual space at random, you'd typically have to do a disk read and write for each new reference and virtual memory would be as slow as a disk. But because programs do actually exhibit strong locality, your operating system can do relatively few swaps per reference.

It's been found by experience that the most effective method for a broad class of memory-usage patterns is very simple; it's called LRU or the "least recently used" algorithm. The virtual-memory system grabs disk blocks into its working set as it needs them. When it runs out of physical memory for the working set, it dumps the least-recently-used block. All Unixes, and most other virtual-memory operating systems, use minor variations on LRU.

Virtual memory is the first link in the bridge between disk and processor speeds. It's explicitly managed by the OS. But there is still a major gap between the speed of physical main memory and the speed at which a processor can access its register memory. The external and internal caches address this, using a technique similar to virtual memory as I've described it.

Just as the physical main memory behaves like a set of windows or caches on the disk's swap area, the external cache acts as windows on main memory. External cache is faster (250M accesses per sec, rather than 100M) and smaller. The hardware (specifically, your computer's memory controller) does the LRU thing in the external cache on blocks of data fetched from the main memory. For historical reasons, the unit of cache swapping is called a line rather than a page.

But we're not done. The internal cache gives us the final step-up in effective speed by caching portions of the external cache. It is faster and smaller yet — in fact, it lives right on the processor chip.

If you want to make your programs really fast, it's useful to know these details. Your programs get faster when they have stronger locality, because that makes the caching work better. The easiest way to make programs fast is therefore to make them small. If a program isn't slowed down by lots of disk I/O or waits on network events, it will usually run at the speed of the smallest cache that it will fit inside.

If you can't make your whole program small, some effort to tune the speed-critical portions so they have stronger locality can pay off. Details on techniques for doing such tuning are beyond the scope of this tutorial; by the time you need them, you'll be intimate enough with some compiler to figure out many of them yourself.
8.3. The Memory Management Unit

Even when you have enough physical core to avoid swapping, the part of the operating system called the memory manager still has important work to do. It has to make sure that programs can only alter their own data segments — that is, prevent erroneous or malicious code in one program from garbaging the data in another. To do this, it keeps a table of data and code segments. The table is updated whenever a process either requests more memory or releases memory (the latter usually when it exits).

This table is used to pass commands to a specialized part of the underlying hardware called an MMU or memory management unit. Modern processor chips have MMUs built right onto them. The MMU has the special ability to put fences around areas of memory, so an out-of-bound reference will be refused and cause a special interrupt to be raised.

If you ever see a Unix message that says "Segmentation fault", "core dumped" or something similar, this is exactly what has happened; an attempt by the running program to access memory (core) outside its segment has raised a fatal interrupt. This indicates a bug in the program code; the core dump it leaves behind is diagnostic information intended to help a programmer track it down.

There is another aspect to protecting processes from each other besides segregating the memory they access. You also want to be able to control their file accesses so a buggy or malicious program can't corrupt critical pieces of the system. This is why Unix has file permissions which we'll discuss later.


9. How does my computer store things in memory?

You probably know that everything on a computer is stored as strings of bits (binary digits; you can think of them as lots of little on-off switches). Here we'll explain how those bits are used to represent the letters and numbers that your computer is crunching.

Before we can go into this, you need to understand about the word size of your computer. The word size is the computer's preferred size for moving units of information around; technically it's the width of your processor's registers, which are the holding areas your processor uses to do arithmetic and logical calculations. When people write about computers having bit sizes (calling them, say, "32-bit" or "64-bit" computers), this is what they mean.

Most computers (including 386, 486, and Pentium PCs) have a word size of 32 bits. The old 286 machines had a word size of 16. Old-style mainframes often had 36-bit words. The AMD Opteron, Intel Itanium, and the Alpha from what used to be DEC and is now Compaq have 64-bit words.

The computer views your memory as a sequence of words numbered from zero up to some large value dependent on your memory size. That value is limited by your word size, which is why programs on older machines like 286s had to go through painful contortions to address large amounts of memory. I won't describe them here; they still give older programmers nightmares.
9.1. Numbers

Integer numbers are represented as either words or pairs of words, depending on your processor's word size. One 32-bit machine word is the most common integer representation.

Integer arithmetic is close to but not actually mathematical base-two. The low-order bit is 1, next 2, then 4 and so forth as in pure binary. But signed numbers are represented in twos-complement notation. The highest-order bit is a sign bit which makes the quantity negative, and every negative number can be obtained from the corresponding positive value by inverting all the bits and adding one. This is why integers on a 32-bit machine have the range -231 to 231 - 1. That 32nd bit is being used for sign; 0 means a positive number or zero, 1 a negative number.

Some computer languages give you access to unsigned arithmetic which is straight base 2 with zero and positive numbers only.

Most processors and some languages can do operations in floating-point numbers (this capability is built into all recent processor chips). Floating-point numbers give you a much wider range of values than integers and let you express fractions. The ways in which this is done vary and are rather too complicated to discuss in detail here, but the general idea is much like so-called ‘scientific notation’, where one might write (say) 1.234 * 1023; the encoding of the number is split into a mantissa (1.234) and the exponent part (23) for the power-of-ten multiplier (which means the number multiplied out would have 20 zeros on it, 23 minus the three decimal places).
9.2. Characters

Characters are normally represented as strings of seven bits each in an encoding called ASCII (American Standard Code for Information Interchange). On modern machines, each of the 128 ASCII characters is the low seven bits of an octet or 8-bit byte; octets are packed into memory words so that (for example) a six-character string only takes up two memory words. For an ASCII code chart, type ‘man 7 ascii’ at your Unix prompt.

The preceding paragraph was misleading in two ways. The minor one is that the term ‘octet’ is formally correct but seldom actually used; most people refer to an octet as byte and expect bytes to be eight bits long. Strictly speaking, the term ‘byte’ is more general; there used to be, for example, 36-bit machines with 9-bit bytes (though there probably never will be again).

The major one is that not all the world uses ASCII. In fact, much of the world can't — ASCII, while fine for American English, lacks many accented and other special characters needed by users of other languages. Even British English has trouble with the lack of a pound-currency sign.

There have been several attempts to fix this problem. All use the extra high bit that ASCII doesn't, making it the low half of a 256-character set. The most widely-used of these is the so-called ‘Latin-1’ character set (more formally called ISO 8859-1). This is the default character set for Linux, HTML, and X. Microsoft Windows uses a mutant version of Latin-1 that adds a bunch of characters such as right and left double quotes in places proper Latin-1 leaves unassigned for historical reasons (for a scathing account of the trouble this causes, see the demoroniser page).

Latin-1 handles western European languages, including English, French, German, Spanish, Italian, Dutch, Norwegian, Swedish, Danish. However, this isn't good enough either, and as a result there is a whole series of Latin-2 through -9 character sets to handle things like Greek, Arabic, Hebrew, Esperanto, and Serbo-Croatian. For details, see the ISO alphabet soup page.

The ultimate solution is a huge standard called Unicode (and its identical twin ISO/IEC 10646-1:1993). Unicode is identical to Latin-1 in its lowest 256 slots. Above these in 16-bit space it includes Greek, Cyrillic, Armenian, Hebrew, Arabic, Devanagari, Bengali, Gurmukhi, Gujarati, Oriya, Tamil, Telugu, Kannada, Malayalam, Thai, Lao, Georgian, Tibetan, Japanese Kana, the complete set of modern Korean Hangul, and a unified set of Chinese/Japanese/Korean (CJK) ideographs. For details, see the Unicode Home Page.

10. How does my computer store things on disk?

When you look at a hard disk under Unix, you see a tree of named directories and files. Normally you won't need to look any deeper than that, but it does become useful to know what's going on underneath if you have a disk crash and need to try to salvage files. Unfortunately, there's no good way to describe disk organization from the file level downwards, so I'll have to describe it from the hardware up.
10.1. Low-level disk and file system structure

The surface area of your disk, where it stores data, is divided up something like a dartboard — into circular tracks which are then pie-sliced into sectors. Because tracks near the outer edge have more area than those close to the spindle at the center of the disk, the outer tracks have more sector slices in them than the inner ones. Each sector (or disk block) has the same size, which under modern Unixes is generally 1 binary K (1024 8-bit words). Each disk block has a unique address or disk block number.

Unix divides the disk into disk partitions. Each partition is a continuous span of blocks that's used separately from any other partition, either as a file system or as swap space. The original reasons for partitions had to do with crash recovery in a world of much slower and more error-prone disks; the boundaries between them reduce the fraction of your disk likely to become inaccessible or corrupted by a random bad spot on the disk. Nowadays, it's more important that partitions can be declared read-only (preventing an intruder from modifying critical system files) or shared over a network through various means we won't discuss here. The lowest-numbered partition on a disk is often treated specially, as a boot partition where you can put a kernel to be booted.

Each partition is either swap space (used to implement virtual memory) or a file system used to hold files. Swap-space partitions are just treated as a linear sequence of blocks. File systems, on the other hand, need a way to map file names to sequences of disk blocks. Because files grow, shrink, and change over time, a file's data blocks will not be a linear sequence but may be scattered all over its partition (from wherever the operating system can find a free block when it needs one). This scattering effect is called fragmentation.
10.2. File names and directories

Within each file system, the mapping from names to blocks is handled through a structure called an i-node. There's a pool of these things near the "bottom" (lowest-numbered blocks) of each file system (the very lowest ones are used for housekeeping and labeling purposes we won't describe here). Each i-node describes one file. File data blocks (including directories) live above the i-nodes (in higher-numbered blocks).

Every i-node contains a list of the disk block numbers in the file it describes. (Actually this is a half-truth, only correct for small files, but the rest of the details aren't important here.) Note that the i-node does not contain the name of the file.

Names of files live in directory structures. A directory structure just maps names to i-node numbers. This is why, in Unix, a file can have multiple true names (or hard links); they're just multiple directory entries that happen to point to the same i-node.
10.3. Mount points

In the simplest case, your entire Unix file system lives in just one disk partition. While you'll see this arrangement on some small personal Unix systems, it's unusual. More typical is for it to be spread across several disk partitions, possibly on different physical disks. So, for example, your system may have one small partition where the kernel lives, a slightly larger one where OS utilities live, and a much bigger one where user home directories live.

The only partition you'll have access to immediately after system boot is your root partition, which is (almost always) the one you booted from. It holds the root directory of the file system, the top node from which everything else hangs.

The other partitions in the system have to be attached to this root in order for your entire, multiple-partition file system to be accessible. About midway through the boot process, your Unix will make these non-root partitions accessible. It will mount each one onto a directory on the root partition.

For example, if you have a Unix directory called /usr, it is probably a mount point to a partition that contains many programs installed with your Unix but not required during initial boot.
10.4. How a file gets looked up

Now we can look at the file system from the top down. When you open a file (such as, say, /home/esr/WWW/ldp/fundamentals.xml) here is what happens:

Your kernel starts at the root of your Unix file system (in the root partition). It looks for a directory there called ‘home’. Usually ‘home’ is a mount point to a large user partition elsewhere, so it will go there. In the top-level directory structure of that user partition, it will look for a entry called ‘esr’ and extract an i-node number. It will go to that i-node, notice that its associated file data blocks are a directory structure, and look up ‘WWW’. Extracting that i-node, it will go to the corresponding subdirectory and look up ‘ldp’. That will take it to yet another directory i-node. Opening that one, it will find an i-node number for ‘fundamentals.xml’. That i-node is not a directory, but instead holds the list of disk blocks associated with the file.
10.5. File ownership, permissions and security

To keep programs from accidentally or maliciously stepping on data they shouldn't, Unix has permission features. These were originally designed to support timesharing by protecting multiple users on the same machine from each other, back in the days when Unix ran mainly on expensive shared minicomputers.

In order to understand file permissions, you need to recall the description of users and groups in the section What happens when you log in?. Each file has an owning user and an owning group. These are initially those of the file's creator; they can be changed with the programs chown(1) and chgrp(1).

The basic permissions that can be associated with a file are ‘read’ (permission to read data from it), ‘write’ (permission to modify it) and ‘execute’ (permission to run it as a program). Each file has three sets of permissions; one for its owning user, one for any user in its owning group, and one for everyone else. The ‘privileges’ you get when you log in are just the ability to do read, write, and execute on those files for which the permission bits match your user ID or one of the groups you are in, or files that have been made accessible to the world.

To see how these may interact and how Unix displays them, let's look at some file listings on a hypothetical Unix system. Here's one:

snark:~$ ls -l notes
-rw-r--r-- 1 esr users 2993 Jun 17 11:00 notes

This is an ordinary data file. The listing tells us that it's owned by the user ‘esr’ and was created with the owning group ‘users’. Probably the machine we're on puts every ordinary user in this group by default; other groups you commonly see on timesharing machines are ‘staff’, ‘admin’, or ‘wheel’ (for obvious reasons, groups are not very important on single-user workstations or PCs). Your Unix may use a different default group, perhaps one named after your user ID.

The string ‘-rw-r--r--’ represents the permission bits for the file. The very first dash is the position for the directory bit; it would show ‘d’ if the file were a directory, or would show ‘l’ if the file were a symbolic link. After that, the first three places are user permissions, the second three group permissions, and the third are permissions for others (often called ‘world’ permissions). On this file, the owning user ‘esr’ may read or write the file, other people in the ‘users’ group may read it, and everybody else in the world may read it. This is a pretty typical set of permissions for an ordinary data file.

Now let's look at a file with very different permissions. This file is GCC, the GNU C compiler.

snark:~$ ls -l /usr/bin/gcc
-rwxr-xr-x 3 root bin 64796 Mar 21 16:41 /usr/bin/gcc

This file belongs to a user called ‘root’ and a group called ‘bin’; it can be written (modified) only by root, but read or executed by anyone. This is a typical ownership and set of permissions for a pre-installed system command. The ‘bin’ group exists on some Unixes to group together system commands (the name is a historical relic, short for ‘binary’). Your Unix might use a ‘root’ group instead (not quite the same as the ‘root' user!).

The ‘root’ user is the conventional name for numeric user ID 0, a special, privileged account that can override all privileges. Root access is useful but dangerous; a typing mistake while you're logged in as root can clobber critical system files that the same command executed from an ordinary user account could not touch.

Because the root account is so powerful, access to it should be guarded very carefully. Your root password is the single most critical piece of security information on your system, and it is what any crackers and intruders who ever come after you will be trying to get.

About passwords: Don't write them down — and don't pick a passwords that can easily be guessed, like the first name of your girlfriend/boyfriend/spouse. This is an astonishingly common bad practice that helps crackers no end. In general, don't pick any word in the dictionary; there are programs called dictionary crackers that look for likely passwords by running through word lists of common choices. A good technique is to pick a combination consisting of a word, a digit, and another word, such as ‘shark6cider’ or ‘jump3joy’; that will make the search space too large for a dictionary cracker. Don't use these examples, though — crackers might expect that after reading this document and put them in their dictionaries.

Now let's look at a third case:

snark:~$ ls -ld ~
drwxr-xr-x 89 esr users 9216 Jun 27 11:29 /home2/esr
snark:~$

This file is a directory (note the ‘d’ in the first permissions slot). We see that it can be written only by esr, but read and executed by anybody else.

Read permission gives you the ability to list the directory — that is, to see the names of files and directories it contains. Write permission gives you the ability to create and delete files in the directory. If you remember that the directory includes a list of the names of the files and subdirectories it contains, these rules will make sense.

Execute permission on a directory means you can get through the directory to open the files and directories below it. In effect, it gives you permission to access the i-nodes in the directory. A directory with execute completely turned off would be useless.

Occasionally you'll see a directory that is world-executable but not world-readable; this means a random user can get to files and directories beneath it, but only by knowing their exact names (the directory cannot be listed).

It's important to remember that read, write, or execute permission on a directory is independent of the permissions on the files and directories beneath. In particular, write access on a directory means you can create new files or delete existing files there, but does not automatically give you write access to existing files.

Finally, let's look at the permissions of the login program itself.

snark:~$ ls -l /bin/login
-rwsr-xr-x 1 root bin 20164 Apr 17 12:57 /bin/login

This has the permissions we'd expect for a system command — except for that ‘s’ where the owner-execute bit ought to be. This is the visible manifestation of a special permission called the ‘set-user-id’ or setuid bit.

The setuid bit is normally attached to programs that need to give ordinary users the privileges of root, but in a controlled way. When it is set on an executable program, you get the privileges of the owner of that program file while the program is running on your behalf, whether or not they match your own.

Like the root account itself, setuid programs are useful but dangerous. Anyone who can subvert or modify a setuid program owned by root can use it to spawn a shell with root privileges. For this reason, opening a file to write it automatically turns off its setuid bit on most Unixes. Many attacks on Unix security try to exploit bugs in setuid programs in order to subvert them. Security-conscious system administrators are therefore extra-careful about these programs and reluctant to install new ones.

There are a couple of important details we glossed over when discussing permissions above; namely, how the owning group and permissions are assigned when a file or directory is first created. The group is an issue because users can be members of multiple groups, but one of them (specified in the user's /etc/passwd entry) is the user's default group and will normally own files created by the user.

The story with initial permission bits is a little more complicated. A program that creates a file will normally specify the permissions it is to start with. But these will be modified by a variable in the user's environment called the umask. The umask specifies which permission bits to turn off when creating a file; the most common value, and the default on most systems, is -------w- or 002, which turns off the world-write bit. See the documentation of the umask command on your shell's manual page for details.

Initial directory group is also a bit complicated. On some Unixes a new directory gets the default group of the creating user (this in the System V convention); on others, it gets the owning group of the parent directory in which it's created (this is the BSD convention). On some modern Unixes, including Linux, the latter behavior can be selected by setting the set-group-ID on the directory (chmod g+s).
10.6. How things can go wrong

Earlier it was hinted that file systems can be fragile things. Now we know that to get to a file you have to hopscotch through what may be an arbitrarily long chain of directory and i-node references. Now suppose your hard disk develops a bad spot?

If you're lucky, it will only trash some file data. If you're unlucky, it could corrupt a directory structure or i-node number and leave an entire subtree of your system hanging in limbo — or, worse, result in a corrupted structure that points multiple ways at the same disk block or i-node. Such corruption can be spread by normal file operations, trashing data that was not in the original bad spot.

Fortunately, this kind of contingency has become quite uncommon as disk hardware has become more reliable. Still, it means that your Unix will want to integrity-check the file system periodically to make sure nothing is amiss. Modern Unixes do a fast integrity check on each partition at boot time, just before mounting it. Every few reboots they'll do a much more thorough check that takes a few minutes longer.

If all of this sounds like Unix is terribly complex and failure-prone, it may be reassuring to know that these boot-time checks typically catch and correct normal problems before they become really disastrous. Other operating systems don't have these facilities, which speeds up booting a bit but can leave you much more seriously screwed when attempting to recover by hand (and that's assuming you have a copy of Norton Utilities or whatever in the first place...).

One of the trends in current Unix designs is journalling file systems. These arrange traffic to the disk so that it's guaranteed to be in a consistent state that can be recovered when the system comes back up. This will speed up the boot-time integrity check a lot.


11. How do computer languages work?

We've already discussed how programs are run. Every program ultimately has to execute as a stream of bytes that are instructions in your computer's machine language. But human beings don't deal with machine language very well; doing so has become a rare, black art even among hackers.

Almost all Unix code except a small amount of direct hardware-interface support in the kernel itself is nowadays written in a high-level language. (The ‘high-level’ in this term is a historical relic meant to distinguish these from ‘low-level’ assembler languages, which are basically thin wrappers around machine code.)

There are several different kinds of high-level languages. In order to talk about these, you'll find it useful to bear in mind that the source code of a program (the human-created, editable version) has to go through some kind of translation into machine code that the machine can actually run.
11.1. Compiled languages

The most conventional kind of language is a compiled language. Compiled languages get translated into runnable files of binary machine code by a special program called (logically enough) a compiler. Once the binary has been generated, you can run it directly without looking at the source code again. (Most software is delivered as compiled binaries made from code you don't see.)

Compiled languages tend to give excellent performance and have the most complete access to the OS, but also to be difficult to program in.

C, the language in which Unix itself is written, is by far the most important of these (with its variant C++). FORTRAN is another compiled language still used among engineers and scientists but years older and much more primitive. In the Unix world no other compiled languages are in mainstream use. Outside it, COBOL is very widely used for financial and business software.

There used to be many other compiler languages, but most of them have either gone extinct or are strictly research tools. If you are a new Unix developer using a compiled language, it is overwhelmingly likely to be C or C++.
11.2. Interpreted languages

An interpreted language depends on an interpreter program that reads the source code and translates it on the fly into computations and system calls. The source has to be re-interpreted (and the interpreter present) each time the code is executed.

Interpreted languages tend to be slower than compiled languages, and often have limited access to the underlying operating system and hardware. On the other hand, they tend to be easier to program and more forgiving of coding errors than compiled languages.

Many Unix utilities, including the shell and bc(1) and sed(1) and awk(1), are effectively small interpreted languages. BASICs are usually interpreted. So is Tcl. Historically, the most important interpretive language has been LISP (a major improvement over most of its successors). Today, Unix shells and the Lisp that lives inside the Emacs editor are probably the most important pure interpreted languages.
11.3. P-code languages

Since 1990 a kind of hybrid language that uses both compilation and interpretation has become increasingly important. P-code languages are like compiled languages in that the source is translated to a compact binary form which is what you actually execute, but that form is not machine code. Instead it's pseudocode (or p-code), which is usually a lot simpler but more powerful than a real machine language. When you run the program, you interpret the p-code.

P-code can run nearly as fast as a compiled binary (p-code interpreters can be made quite simple, small and speedy). But p-code languages can keep the flexibility and power of a good interpreter.

Important p-code languages include Python, Perl, and Java.


12. How does the Internet work?

To help you understand how the Internet works, we'll look at the things that happen when you do a typical Internet operation — pointing a browser at the front page of this document at its home on the Web at the Linux Documentation Project. This document is

http://www.tldp.org/HOWTO/Unix-and-Internet-Fundamentals-HOWTO/index.html

which means it lives in the file HOWTO/Unix-and-Internet-Fundamentals-HOWTO/index.html under the World Wide Web export directory of the host www.tldp.org.
12.1. Names and locations

The first thing your browser has to do is to establish a network connection to the machine where the document lives. To do that, it first has to find the network location of the host www.tldp.org (‘host’ is short for ‘host machine’ or ‘network host'; www.tldp.org is a typical hostname). The corresponding location is actually a number called an IP address (we'll explain the ‘IP’ part of this term later).

To do this, your browser queries a program called a name server. The name server may live on your machine, but it's more likely to run on a service machine that yours talks to. When you sign up with an ISP, part of your setup procedure will almost certainly involve telling your Internet software the IP address of a nameserver on the ISP's network.

The name servers on different machines talk to each other, exchanging and keeping up to date all the information needed to resolve hostnames (map them to IP addresses). Your nameserver may query three or four different sites across the network in the process of resolving www.tldp.org, but this usually happens very quickly (as in less than a second). We'll look at how nameservers detail in the next section.

The nameserver will tell your browser that www.tldp.org's IP address is 152.19.254.81; knowing this, your machine will be able to exchange bits with www.tldp.org directly.
12.2. The Domain Name System

The whole network of programs and databases that cooperates to translate hostnames to IP addresses is called ‘DNS’ (Domain Name System). When you see references to a ‘DNS server’, that means what we just called a nameserver. Now I'll explain how the overall system works.

Internet hostnames are composed of parts separated by dots. A domain is a collection of machines that share a common name suffix. Domains can live inside other domains. For example, the machine www.tldp.org lives in the .tldp.org subdomain of the .org domain.

Each domain is defined by an authoritative name server that knows the IP addresses of the other machines in the domain. The authoritative (or ‘primary') name server may have backups in case it goes down; if you see references to a secondary name server or (‘secondary DNS') it's talking about one of those. These secondaries typically refresh their information from their primaries every few hours, so a change made to the hostname-to-IP mapping on the primary will automatically be propagated.

Now here's the important part. The nameservers for a domain do not have to know the locations of all the machines in other domains (including their own subdomains); they only have to know the location of the nameservers. In our example, the authoritative name server for the .org domain knows the IP address of the nameserver for .tldp.org but not the address of all the other machines in .tldp.org.

The domains in the DNS system are arranged like a big inverted tree. At the top are the root servers. Everybody knows the IP addresses of the root servers; they're wired into your DNS software. The root servers know the IP addresses of the nameservers for the top-level domains like .com and .org, but not the addresses of machines inside those domains. Each top-level domain server knows where the nameservers for the domains directly beneath it are, and so forth.

DNS is carefully designed so that each machine can get away with the minimum amount of knowledge it needs to have about the shape of the tree, and local changes to subtrees can be made simply by changing one authoritative server's database of name-to-IP-address mappings.

When you query for the IP address of www.tldp.org, what actually happens is this: First, your nameserver asks a root server to tell it where it can find a nameserver for .org. Once it knows that, it then asks the .org server to tell it the IP address of a .tldp.org nameserver. Once it has that, it asks the .tldp.org nameserver to tell it the address of the host www.tldp.org.

Most of the time, your nameserver doesn't actually have to work that hard. Nameservers do a lot of cacheing; when yours resolves a hostname, it keeps the association with the resulting IP address around in memory for a while. This is why, when you surf to a new website, you'll usually only see a message from your browser about "Looking up" the host for the first page you fetch. Eventually the name-to-address mapping expires and your DNS has to re-query — this is important so you don't have invalid information hanging around forever when a hostname changes addresses. Your cached IP address for a site is also thrown out if the host is unreachable.
12.3. Packets and routers

What the browser wants to do is send a command to the Web server on www.tldp.org that looks like this:

GET /LDP/HOWTO/Fundamentals.html HTTP/1.0

Here's how that happens. The command is made into a packet, a block of bits like a telegram that is wrapped with three important things; the source address (the IP address of your machine), the destination address (152.19.254.81), and a service number or port number (80, in this case) that indicates that it's a World Wide Web request.

Your machine then ships the packet down the wire (your connection to your ISP, or local network) until it gets to a specialized machine called a router. The router has a map of the Internet in its memory — not always a complete one, but one that completely describes your network neighborhood and knows how to get to the routers for other neighborhoods on the Internet.

Your packet may pass through several routers on the way to its destination. Routers are smart. They watch how long it takes for other routers to acknowledge having received a packet. They also use that information to direct traffic over fast links. They use it to notice when another router (or a cable) have dropped off the network, and compensate if possible by finding another route.

There's an urban legend that the Internet was designed to survive nuclear war. This is not true, but the Internet's design is extremely good at getting reliable performance out of flaky hardware in an uncertain world. This is directly due to the fact that its intelligence is distributed through thousands of routers rather than concentrated in a few massive and vulnerable switches (like the phone network). This means that failures tend to be well localized and the network can route around them.

Once your packet gets to its destination machine, that machine uses the service number to feed the packet to the web server. The web server can tell where to reply to by looking at the command packet's source IP address. When the web server returns this document, it will be broken up into a number of packets. The size of the packets will vary according to the transmission media in the network and the type of service.
12.4. TCP and IP

To understand how multiple-packet transmissions are handled, you need to know that the Internet actually uses two protocols, stacked one on top of the other.

The lower level, IP (Internet Protocol), is responsible for labeling individual packets with the source address and destination address of two computers exchanging information over a network. For example, when you access http://www.tldp.org, the packets you send will have your computer's IP address, such as 192.168.1.101, and the IP address of the www.tldp.org computer, 152.2.210.81. These addresses work in much the same way that your home address works when someone sends you a letter. The post office can read the address and determine where you are and how best to route the letter to you, much like a router does for Internet traffic.

The upper level, TCP (Transmission Control Protocol), gives you reliability. When two machines negotiate a TCP connection (which they do using IP), the receiver knows to send acknowledgements of the packets it sees back to the sender. If the sender doesn't see an acknowledgement for a packet within some timeout period, it resends that packet. Furthermore, the sender gives each TCP packet a sequence number, which the receiver can use to reassemble packets in case they show up out of order. (This can easily happen if network links go up or down during a connection.)

TCP/IP packets also contain a checksum to enable detection of data corrupted by bad links. (The checksum is computed from the rest of the packet in such a way that if either the rest of the packet or the checksum is corrupted, redoing the computation and comparing is very likely to indicate an error.) So, from the point of view of anyone using TCP/IP and nameservers, it looks like a reliable way to pass streams of bytes between hostname/service-number pairs. People who write network protocols almost never have to think about all the packetizing, packet reassembly, error checking, checksumming, and retransmission that goes on below that level.
12.5. HTTP, an application protocol

Now let's get back to our example. Web browsers and servers speak an application protocol that runs on top of TCP/IP, using it simply as a way to pass strings of bytes back and forth. This protocol is called HTTP (Hyper-Text Transfer Protocol) and we've already seen one command in it — the GET shown above.

When the GET command goes to www.tldp.org's webserver with service number 80, it will be dispatched to a server daemon listening on port 80. Most Internet services are implemented by server daemons that do nothing but wait on ports, watching for and executing incoming commands.

If the design of the Internet has one overall rule, it's that all the parts should be as simple and human-accessible as possible. HTTP, and its relatives (like the Simple Mail Transfer Protocol, SMTP, that is used to move electronic mail between hosts) tend to use simple printable-text commands that end with a carriage-return/line feed.

This is marginally inefficient; in some circumstances you could get more speed by using a tightly-coded binary protocol. But experience has shown that the benefits of having commands be easy for human beings to describe and understand outweigh any marginal gain in efficiency that you might get at the cost of making things tricky and opaque.

Therefore, what the server daemon ships back to you via TCP/IP is also text. The beginning of the response will look something like this (a few headers have been suppressed):

HTTP/1.1 200 OK
Date: Sat, 10 Oct 1998 18:43:35 GMT
Server: Apache/1.2.6 Red Hat
Last-Modified: Thu, 27 Aug 1998 17:55:15 GMT
Content-Length: 2982
Content-Type: text/html

These headers will be followed by a blank line and the text of the web page (after which the connection is dropped). Your browser just displays that page. The headers tell it how (in particular, the Content-Type header tells it the returned data is really HTML).


13. To Learn More

There is a Reading List HOWTO that lists books you can read to learn more about the topics we have touched on here. You might also want to read the How To Become A Hacker document.



v 0.3 of J. Random Hacker
ON HACKING ITSELF
According to RMS, it is difficult to pin down the exact nature of a hack (he's considered something of an authority on these matters). A few properties of a hack always hold true: cleverness, playfulness, and exploration. This file is about those who self


-identify as hackers, and they mostly program and tinker with computers.

To be exact, a hacker is "a person who creates and modifies computer software and computer hardware, including computer programming, administration, and security."

Negatively viewed hackers include black-hats, software crackers, and phreaks.

* Phreaks, like hackers, are not always "evil." The Woz was a phreak. Phreaks may just have a telecommunications obsession.
* Black-hats are the hackers you'll typically hear about on television, the "bad" ones.
* Grey and white hats are generally just after information or exploring to help find security flaws.
* Software crackers just plain aren't hackers (at least if they illegaly distribute the software).



General Appearance
There isn't a general hacker appearance. Why? Because there isn't one nationality, there isn't one race, there isn't one sex, and there isn't one style of a hacker. Since hackers come from all over the world, the same applies. But again, the media helps
poison the minds of the young and old, the smart and the stupid by propagating the idea of what a hacker looks like. They say a hacker is usually a heavy man, with glasses, bald(ing) head, lonely, and typing away at a computer. Wrong! There's not a specif

ic
style that this culture lays claim to.
around age of 16 to 25

Dress
Just t-shirts and jeans, and a sweatshirt/jacket when necessary. Hackers really despise dressing up. Why the hatred for dressing up? It takes time away from important things and involves conforming. It must be bad.

Reading Habits
Besides technical manuals, hackers do tend to read lots of science fiction. Some specific books that seem to show up on various hacker cultural examinations are Neal Stephenson's Snow Crash (Stephenson was a programmer), William Gibson's Neuromancer, and
Douglas Adams' Hitchiker's Guide to the Galaxy. Outside of the science fiction realm, Godel Escher Bach: An Eternal Golden Braid by Douglas R. Hofstadter and Stephen Levy's Hackers are fairly common. The only really required fantasy series is Lord of the
Rings. Apparently hackers like fantasy, although science fiction is definitely the dominant form of fiction. A hacker often just reads things that expand their intellect. Some subjects include hard science, philosophy, mathematics, technology in general,
and perhaps some history and social science (although the social sciences are a bit fickle. quantitative data is more respect-worthy than qualitative). The webpage slashdot deserves special mention. Keeping slashdot karma up may qualify as an "interest."
There are some competitors emerging in the geek news site arena, digg being the real stand-out.

Other Interests
Following news probably is a hacker interest (hacker news, at any rate). Topics related to FOSS, Linux, BSD, and other off-shoots of the programmer/hacker community are popular. Hardware hacking, case modding, system administration, robotics, and general
technology, of course.Privacy, encryption, and security are all obsessions of hackers, and privacy basically amounts to the only political issue hackers are deeply concerned with (besides anti-trust laws), at least in most cases. Developing general knowle
dge
(see reading habits). Keeping up to date on science advances and science fiction releases seem to be the other main hobbies. Gaming is abundant, especially first-person shooters, real-time strategy games, puzzle games, classic/emulated games, and role










playing games (especially MMORPGs, but some Eastern console RPGs and a few MUDs are likely to be on a hacker's list of frequently played games. A few Western single player RPGs are played as well, mostly classics such as Zork and nethack). Hackers lean to











wards games they can hack, of course (not in the black hat manner, but games easily modifiable. This needs to be said because black-hat hacking is a major issue in gaming, and has taken on the name "hacking"). Some games especially popular among hackers a











re the (not completely) open source games Quake, Wolfenstein, and Doom; also Unreal Tournament, Half-Life, and World of Warcraft, though not all hackers play those games because they are not OSS.
The relation between hackers and hobby games isn't really there, except for roleplaying games and board games. Miniature wargames and collectible card games are very impractical, and not played as often (often the winner is the one who has spent more of h











is money, especially in CCGs. Miniature wargames take long amounts of time away from hacking and in order to play you have to glue and paint dozens of little men, often in identical fashion. Its incredibly wasteful, especially in these days when you can s











imply download Freeciv or something). A few hackers also will LARP/go to SCA meetings on occasion, but not if it involves a lot of make-up or impractical clothing. Basically hackers often play "mind sports" like settlers of catan, chess, draughts, and go,











along with D&D or some other system, and perhaps some sort of brain-teasing puzzle. Hackers love puzzles, and favorites include PopCap Games games and Rubik's cubes.Anime deserves mention in this section, especially science-fiction classics such as Ghost
In The Shell, Cowboy Bebop, and Neon Genesis Evangelion. Hackers are likely to read the latest updates to various webcomics and perhaps some real graphic novel/mangastuff. Prime examples of loved movies/shows include: Lord of the Rings, Star Wars, Star Tr











ek, and The Matrix. Babylon 5 also seems to be imporant, and there are plenty of hit sci-fi shows that hackers enjoy. Monty Python is beloved of hackers, and they can often recite sketches for you. Hackers look down on newbie Pythonites who can only quote











The Holy Grail. Hacker humor involves showing cleverness, especially clever use of language, and subtle intellectual currents. Pranks are always amusing and reversible. Looney Tunes may be a surprising example of comedy hackers enjoy.

Hackers love music. Music that hackers enjoy is likely to be complex or just very good. They enjoy most genres of music (pop, R&B, hip-hop, and popular rock being exceptions). Hackers will often have anime or science fiction soundtracks. They are likely t











o play instruments that involve lots of gadgetry, such as (but definetely not limited to) electric guitar and bass.
Take note outsiders: Hackers aren't obsessed with only math, science, and engineering. They have strong minds for the humanities as well (digital audio, video, photagraphy, design, graphics, music, roleplaying, case modding, fan fictions, and the list goe











s on!).


Education
Often attend college for at least a bachelor's, especially in current times when employers are very biased against autodidacts. Most common majors will be computer science and electrical engineering, and less commonly computer engineering (sub-branch of E












E), IT/Information Systems, and software engineering. Those are all very interelated subjects. Artificial Intelligence, Cognitive Science, physics, and linguistics are disciplines computer science research often crosses into, according to wikipedia.
Hackers will often just study challenging and interesting courses that cover a little humanities (philosophy, arts related to their other interests) and a little social science (anything heavily statistical and scientific, linguistics), besides all of the











math and science and engineering, which will dominate their courses.

Some of the best universites for hackers are the following: MIT, Carnegie Mellon, Berkeley, Stanford, and Illinois-Urbana-Champaign (they are supposedly the best colleges for EE/CS/EECS)

Things Hackers Detest and Avoid
Dressing up. Vocal religious extremists. Vocal and not-too-bright people. Microsoft. Poor spellers. Republicans. Democrats. AOL users. RIAA. Script kiddies. Whiners. Hacker wannabes. Spoiled little brats, Bureacracy, bad music
and popular television (with a few exceptions, see other interests). Boredom, dishonesty, etc. COBOL, BASIC, and FORTRAN.

Food
Quick and unusual. Quick will usually win the day. Pizza. Ramen. Junk food in general. Mexican food, Oriental food(mostly chinese take-out).

Politics
Anti-authoritarianism. Anti-DRM(Data Rights Management; advanced during the Clinton Adminstration and is the reason you can go to jail for making backup copies of your DVDs and various other forms of electronic media)

Gender and Ethnicity
Commonly White/Jewish/Asian males. Hackers do want diversification, but for socio-economic reasons most of America's races have not caught up with those three. No true hacker is openly prejudiced, and those who are inwardly prejudiced fight it (that would











include all hackers). There is a small amount of females, but most women have, unfortunately, developed a fear of computers. Hackers want this stuff to change, but don't go around making people feel guilty about sociological processes that are beyond
their control.

On a controversial note (perhaps worthy of ESR), Jews and Asians have cultures that (stereotypically) value study. Practicing Jews spend lots of time studying religious texts, and Japanese students have a very high rate of suicide. This isn't to say that
other races may not value study, but Judaism requires it, and so does Asian society. Of course hackers want diversification, and realize that there are a lot of stupid Jews, Asians, Whites, and people in general.

Religion
Atheism, non-believing, neo-pagans, and a few parody religions, with some mainstream religions included. Christian, Catholic, and Jewish hackers are not unknown. Hackers do however want diversity and don't supress religeous practicing that does not limit


their autonomy. Worshipping the pantheon found in a fantasy novel is not uncommon, but usually donewith some sarcasm.

Ceremonial Chemicals
CAFFEINE owns the life of all hackers. Hackers will not take drugs with possible negative effects to brainpower, so no marijuana or speed ("octagons"). Presumably hackers tolerate stimulants and hallucinogens a bit more than other segments of the populati











on. Maybe some of the odd ones who read Wired and Mondo 2000 in the early 90s, listened to industrial, and have seen the Matrix at least five hundred times use nootropics, or "smart drugs."

Communication Style
Somewhat polite, large vocabulary, fairly quiet. Generally can write well. Can give quite the "tongue lashing" to those who are beligerently ignorant and hostile towards them.

Geographical Distribution
Largely in Northeast and Pacific West, although technology jobs are springing up everywhere.

Sexual Habits
Gay and bisexual hackers are more outspoken than in the general population, and tolerated. Hackers are known to have less sex than others, but sexual activity is pretty normal in reality.

Hackers tolorence to other cultures extends to other sexualities
Personality Characteristics
"Weirdos." Possibly lazy, but in general pretty focused. Intelligent, individualistic. Rarely hardcore gothic, emo 'hackers' are rare and shunned. Those who pick verbal fights are usually pwnd; but because of the time spent at the computer hackers dont
have much time to work out and tend to shy away from physical encounters.
To use pseudo-scientific personality theories, hackers are INTPs in Myers-Brigg, and Type 5 in enneagrams. Descriptions of Type 5s and INTPs are nearly the same as the
hacker personality, this isn't just a stereotype.

Weakness of the Hacker Personality
J. Random Hacker doesn't believe the hacker personality has any weaknesses. So it was decided to put "arrogance," and being a little harsh towards "not-so-smart" people.

Miscellaneous
Hackers are very unorganized in their physical world. They have awful handwriting, almost as a rule.
Dont get on the bad side of a hacker, because of their nature they will find it easy to come up with...unorthodox ways of getting revenge.



On Hacking
In June 2000, while visiting Korea, I did a fun hack that clearly illustrates the original and true meaning of the word "hacker".

I went to lunch with some GNU fans, and was sitting down to eat some tteokpaekki (*), when a waitress set down six chopsticks right in front of me. It occurred to me that perhaps these were meant for three people, but it was more amusing to imagine that I was supposed to use all six. I did not know any way to do that, so I realized that if I could come up with a way, it would be a hack. I started thinking. After a few seconds I had an idea.

First I used my left hand to put three chopsticks into my right hand. That was not so hard, though I had to figure out where to put them so that I could control them individually. Then I used my right hand to put the other three chopsticks into my left hand. That was hard, since I had to keep the three chopsticks already in my right hand from falling out. After a couple of tries I got it done.

Then I had to figure out how to use the six chopsticks. That was harder. I did not manage well with the left hand, but I succeeded in manipulating all three in the right hand. After a couple of minutes of practice and adjustment, I managed to pick up a piece of food using three sticks converging on it from three different directions, and put it in my mouth.

It didn't become easy--for practical purposes, using two chopsticks is completely superior. But precisely because using three in one hand is hard and ordinarily never thought of, it has "hack value", as my lunch companions immediately recognized. Playfully doing something difficult, whether useful or not, that is hacking.

I later told the Korea story to a friend in Boston, who proceded to put four chopsticks in one hand and use them as two pairs--picking up two different pieces of food at once, one with each pair. He had topped my hack. Was his action, too, a hack? I think so. Is he therefore a hacker? That depends on how much he likes to hack.

The hacking community developed at MIT and some other universities in the 1960s and 1970s. Hacking included a wide range of activities, from writing software, to practical jokes, to exploring the roofs and tunnels of the MIT campus. Other activities, performed far from MIT and far from computers, also fit hackers' idea of what hacking means: for instance, I think the controversial 1950s "musical piece" by John Cage, 4'33", which has no notes, is more of a hack than a musical composition. The palindromic three-part piece written by Guillaume de Machaut in the 1300s, "Ma Fin Est Mon Commencement", was also a good hack, even better because it also sounds good. Puck appreciated hack value.

It is hard to write a simple definition of something as varied as hacking, but I think what these activities have in common is playfulness, cleverness, and exploration. Thus, hacking means exploring the limits of what is possible, in a spirit of playful cleverness. Activities that display playful cleverness have "hack value".

Hackers typically had little respect for the silly rules that administrators like to impose, so they looked for ways around. For instance, when computers at MIT started to have "security" (that is, restrictions on what users could do), some hackers found clever ways to bypass the security, partly so they could use the computers freely, and partly just for the sake of cleverness (hacking does not need to be useful). However, only some hackers did this--many were occupied with other kinds of cleverness, such as placing some amusing object on top of MIT's great dome (**), finding a way to do a certain computation with only 5 instructions when the shortest known program required 6, writing a program to print numbers in roman numerals, or writing a program to understand questions in English.

Meanwhile, another group of hackers at MIT found a different solution to the problem of computer security: they designed the Incompatible Timesharing System without security "features". In the hacker's paradise, the glory days of the Artificial Intelligence Lab, there was no security breaking, because there was no security to break. It was there, in that environment, that I learned to be a hacker, though I had shown the inclination previously. We had plenty of other domains in which to be playfully clever, without building artificial security obstacles which then had to be overcome.

Yet when I say I am a hacker, people often think I am making a naughty admission, presenting myself specifically as a security breaker. How did this confusion develop?

Around 1980, when the news media took notice of hackers, they fixated on one narrow aspect of real hacking: the security breaking which some hackers occasionally did. They ignored all the rest of hacking, and took the term to mean breaking security, no more and no less. The media have since spread that definition, disregarding our attempts to correct them. As a result, most people have a mistaken idea of what we hackers actually do and what we think.

You can help correct the misunderstanding simply by making a distinction between security breaking and hacking--by using the term "cracking" for security breaking. The people who do it are "crackers". Some of them may also be hackers, just as some of them may be chess players or golfers; most of them are not.

* Pronounced like stuckpeckee without the s, if I recall right.

** Going on the great dome is "forbidden", so in a sense it constitutes "breaking security". Nonetheless, the MIT museum proudly exhibited photos of some of the best dome hacks, as well as some of the objects that hackers placed on the dome in their hacks. The MIT administration thus implicitly recognizes that "breaking security" is not necessarily a monstrous act and should not automatically be excoriated. Whether security breaking is wrong depends on what the security breaker proceeds to *do* with the "forbidden" access thus obtained. Hurting people is bad, amusing the community is good.

@@ This is a picture of a hack that we did in England in the 80s, changing Customer Training College into Customer Draining College by replacing one letter of the sign. You can view all the hacks pictures here.


The Hacker FAQ

The following list is an attempt to cover some of the issues that will invariably come up when people without previous experience of the hacker community try to hire a hacker. This FAQ is intended for free distribution, and may be copied as desired. It is in an early revision. If you wish to modify the FAQ, or distribute it for publication, please contact the author. The author is seebs@plethora.net. The official distribution site (as of revision 0.05) is "http://www.plethora.net/~seebs/faqs/hacker.html".

DISCLAIMER: The author is a hacker. Bias is inevitable.

This document is copyright 1995, 1996, 1998, 1999 Peter Seebach. Unaltered distribution is permitted.

Revision 0.05 - Last modified September 28, 1999
Questions and Answers:
Section 0: Basic understanding.

0.0: Won't my hacker break into my computer and steal my trade secrets?
No. Hackers aren't, contrary to media reporting, the people who break into computers. Those are crackers. Hackers are people who enjoy playing with computers. Your hacker may occasionally circumvent security measures, but this is not malicious; she just does it when the security is in her way, or because she's curious.
0.1: Was it a good idea to hire a hacker?
It depends on the job. A hacker can be dramatically more effective than a non-hacker at a job, or dramatically less effective. Jobs where hackers are particularly good are:

* Systems administration
* Programming
* Design

Jobs where hackers are particularly bad are:

* Data entry

More generally, a job that requires fast and unexpected changes, significant skill, and is not very repetitive will be one a hacker will excel at. Repetitive, simple jobs are a waste of a good hacker, and will make your hacker bored and frustrated. No one works well bored and frustrated.

The good news is, if you get a hacker on something he particularly likes, you will frequently see performance on the order of five to ten times what a "normal" worker would produce. This is not consistent, and you shouldn't expect to see it all the time, but it will happen. This is most visible on particularly difficult tasks.
0.2: How should I manage my hacker?
The same way you herd cats. It can be a bit confusing; they're not like most other workers. Don't worry! Your hacker is likely to be willing to suggest answers to problems, if asked. Most hackers are nearly self-managing.
0.3: Wait, you just said "10 times", didn't you? You're not serious, right?
Actually, I said "ten times". And yes, I am serious; a hacker on a roll may be able to produce, in a period of a few months, something that a small development group (say, 7-8 people) would have a hard time getting together over a year. He also may not. Your mileage will vary.

IBM used to report that certain programmers might be as much as 100 times as productive as other workers, or more. This kind of thing happens.
0.4: I don't understand this at all. This is confusing. Is there a book on this?
Not yet. In the meantime, check out The New Hacker's Dictionary (references below; also known as "the jargon file"), in particular some of the appendices. The entire work is full of clarifications and details of how hackers think.

Section 1: Social issues

1.0: My hacker doesn't fit in well with our corporate society. She seems to do her work well, but she's not really making many friends.
This is common. Your hacker may not have found any people around who get along with hackers. You may wish to consider offering her a position telecommuting, or flexible hours (read: night shift), which may actually improve her productivity. Or hire another one.
1.1: My hacker seems to dress funny. Is there any way to impress upon him the importance of corporate appearance?
Your hacker has a very good understanding of the importance of corporate appearance. It doesn't help you get your job done. IBM, Ford, and Microsoft have all realized that people work better when they can dress however they want. Your hacker is dressed comfortably. A polite request to dress up some for special occasions may well be honored, and most hackers will cheerfully wear clothes without (unintentional) holes in them if specifically asked.
1.2: My hacker won't call me by my title, and doesn't seem to respect me at all.
Your hacker doesn't respect your title. Hackers don't believe that management is "above" engineering; they believe that management is doing one job, and engineering is doing another. They may well frequently talk as if management is beneath them, but this is really quite fair; your question implies that you talk as if engineering is beneath you. Treat your hacker as an equal, and she will probably treat you as an equal -- quite a compliment!
1.3: My hacker constantly insults the work of my other workers.
Take your hacker aside, and ask for details of what's wrong with the existing work. It may be that there's something wrong with it. Don't let the fact that it runs most of the time fool you; your hacker is probably bothered by the fact that it crashes at all. He may be able to suggest improvements which could dramatically improve performance, reliability, or other features. It's worth looking into.

You may be able to convince your hacker to be more polite, but if there appear to be major differences, it's quite possible that one or more of your existing staff are incompetent. Note that hackers, of course, have different standards of competence than many other people. (Read "different" as "much higher".)

Section 2: Productivity.

2.0: My hacker plays video games on company time.
Hackers, writers, and painters all need some amount of time to spend "percolating" -- doing something else to let their subconscious work on a problem. Your hacker is probably stuck on something difficult. Don't worry about it.
2.1: But it's been two weeks since I saw anything!
Your hacker is working, alone probably, on a big project, and just started, right? She's probably trying to figure it all out in advance. Ask her how it's going; if she starts a lot of sentences, but interrupts them all with "no, wait..." or "drat, that won't work", it's going well.
2.2: Isn't this damaging to productivity?
No. Your hacker needs to recreate and think about things in many ways. He will be more productive with this recreation than without it. Your hacker enjoys working; don't worry about things getting done reasonably well and quickly.
2.3: My hacker is constantly doing things unrelated to her job responsibilities.
Do they need to be done? Very few hackers can resist solving a problem when they can solve it, and no one else is solving it. For that matter, is your hacker getting her job done? If so, consider these other things a freebie or perk (for you). Although it may not be conventional, it's probably helping out quite a bit.
2.4: My hacker is writing a book, reading USENET news, playing video games, talking with friends on the phone, and building sculptures out of paper clips. On company time!
He sounds happy. The chances are he's in one of three states:

1. Basic job responsibilities are periodic (phone support, documentation, et al.) and there's a lull in incoming work. Don't worry about it!
2. Your hacker is stuck on a difficult problem.
3. Your hacker is bored silly and is trying to find amusement. Perhaps you should find him more challenging work?

Any of these factors may be involved. All of them may be involved. In general, if the work is challenging, and is getting done, don't worry too much about the process. You might ask for your corporation to be given credit in the book.
2.5: But my other workers are offended by my hacker's success, and it hurts their productivity.
Do you really need to have workers around who would rather be the person getting something done, than have it done already? Ego has very little place in the workplace. If they can't do it well, assign them to something they can do.

Section 3: Stimulus and response

3.0: My hacker did something good, and I want to reward him.
Good! Here are some of the things most hackers would like to receive in exchange for their work:

1. Respect.
2. Admiration.
3. Compliments.
4. Understanding.
5. Discounts on expensive toys.
6. Money.

These are not necessarily in order. The 4th item (understanding) is the most difficult. Try to remember this good thing your hacker just did the next time you discover he just spent a day playing x-trek. Rather than complaining about getting work done, write it off as "a perk" that was granted (informally) as a bonus for a job well done. Don't worry; hackers get bored quickly when they aren't doing their work.
3.1: My hacker did something bad, and I want to punish him.
Don't. 30 years of psychological research has shown that punishment has no desirable long-term effects. Your hacker is not a lab rat. (Even if he were a lab rat, punishment wouldn't work; at least, not if he were one of the sorts of lab rats the psych research was done on.) If you don't like something your hacker is doing, express your concerns. Explain what it is that bothers you about the behavior.

Be prepared for an argument; your hacker is a rational entity, and presumably had reasons. Don't jump on him too quickly; they may turn out to be good reasons.

Don't be afraid to apologize if you're wrong. If your hacker admits to having been wrong, don't demand an apology; so far as the hacker is concerned, admitting to being wrong is an apology, most likely.
3.2: I don't get it. I offered my hacker a significant promotion, and she turned it down and acted offended.
A promotion frequently involves spending more time listening to people describing what they're doing, and less time playing with computers. Your hacker is enjoying her work; if you want to offer a reward, consider an improvement in title, a possible raise, and some compliments. Make sure your hacker knows you are pleased with her accomplishments -- that's what she's there for.
3.3: My company policy won't let me give my hacker any more raises until he's in management.
Your company policy is broken. A hacker can earn as much as $200 an hour (sometimes more) doing freelance consulting. You may wish to offer your hacker a contracted permanent consulting position with benefits, or otherwise find loopholes. Or, find perks to offer - many hackers will cheerfully accept a discount on hardware from their favorite manufacturer as an effective raise.
3.4: I can't believe the hacker on my staff is worth as much as we're paying.
Ask the other staff in the department what the hacker does, and what they think of it. The chances are that your hacker is spending a few hours a week answering arcane questions that would otherwise require an expensive external consultant. Your hacker may be fulfilling another job's worth of responsibilities in his spare time around the office. Very few hackers aren't worth what they're getting paid; they enjoy accomplishing difficult tasks, and improving worker efficiency.

Section 4: What does that mean?

4.0: My hacker doesn't speak English. At least, I don't think so.
Your hacker is a techie. Your best bet is to pick up a copy of TNHD (The New Hacker's Dictionary). It can be found as http://catb.org/esr/jargon (last I checked) or from a good bookstore. If you have trouble understanding that reference, ask your hacker if she has a copy, or would be willing to explain her terms. Most hackers are willing to explain terms. Be ready for condescension; it's not intended as an insult, but if you don't know the words, she probably has to talk down to you at first to explain them.

It's a reasonably difficult set of words; there are a lot of them, and their usage is much more precise than it sounds. Hackers love word games.

[It is also possible that English is not your hacker's native language, and that it's not yours either. Feel free to substitute a more appropriate language.]
4.1: I can't get an estimate out of my hacker.
Your hacker hasn't figured out how hard the problem is yet. Unlike most workers, hackers will try very hard to refuse to give an estimate until they know for sure that they understand the problem. This may include solving it.

No good engineer goes beyond 95% certainty. Most hackers are good engineers. If you say you will not try to hold him to the estimate (and mean it!) you are much more likely to get an approximate estimate. The estimate may sound very high or very low; it may be very high or very low. Still, it's an estimate, and you get what you ask for.
4.2: My hacker makes obscure, meaningless jokes.
If you feel brave, ask for an explanation. Most of them can be explained. It may take a while, but it may prove interesting.
4.3: My hacker counts from zero.
So does the computer. You can hide it, but computers count from zero. Most hackers do by habit, also.


How to be a Programmer: A Short, Comprehensive, and Personal Summary
Robert L Read

Copyright © 2002, 2003 Robert L. Read
Copyright

Copyright © 2002, 2003
by Robert L. Read. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with one Invariant Section being ‘History (As of February, 2003)’, no Front-Cover Texts, and one Back-Cover Text: ‘The original version of this document was written by Robert L. Read without renumeration and dedicated to the programmers of Hire.com.’ A copy of the license is included in the section entitled ‘GNU Free Documentation License’.

2002
Dedication

To the programmers of Hire.com.

Table of Contents

1. Introduction

2. Beginner

Personal Skills

Learn to Debug
How to Debug by Splitting the Problem Space
How to Remove an Error
How to Debug Using a Log
How to Understand Performance Problems
How to Fix Performance Problems
How to Optimize Loops
How to Deal with I/O Expense
How to Manage Memory
How to Deal with Intermittent Bugs
How to Learn Design Skills
How to Conduct Experiments

Team Skills

Why Estimation is Important
How to Estimate Programming Time
How to Find Out Information
How to Utilize People as Information Sources
How to Document Wisely
How to Work with Poor Code
How to Use Source Code Control
How to Unit Test
Take Breaks when Stumped
How to Recognize When to Go Home
How to Deal with Difficult People

3. Intermediate

Personal Skills

How to Stay Motivated
How to be Widely Trusted
How to Tradeoff Time vs. Space
How to Stress Test
How to Balance Brevity and Abstraction
How to Learn New Skills
Learn to Type
How to Do Integration Testing
Communication Languages
Heavy Tools
How to analyze data

Team Skills

How to Manage Development Time
How to Manage Third-Party Software Risks
How to Manage Consultants
How to Communicate the Right Amount
How to Disagree Honestly and Get Away with It

Judgement

How to Tradeoff Quality Against Development Time
How to Manage Software System Dependence
How to Decide if Software is Too Immature
How to Make a Buy vs. Build Decision
How to Grow Professionally
How to Evaluate Interviewees
How to Know When to Apply Fancy Computer Science
How to Talk to Non-Engineers

4. Advanced

Technological Judgment

How to Tell the Hard From the Impossible
How to Utilize Embedded Languages
Choosing Languages

Compromising Wisely

How to Fight Schedule Pressure
How to Understand the User
How to Get a Promotion

Serving Your Team

How to Develop Talent
How to Choose What to Work On
How to Get the Most From Your Teammates
How to Divide Problems Up
How to Handle Boring Tasks
How to Gather Support for a Project
How to Grow a System
How to Communicate Well
How to Tell People Things They Don't Want to Hear
How to Deal with Managerial Myths
How to Deal with Organizational Chaos

Glossary
A.
B. History (As Of February, 2003)
C. GNU Free Documentation License

PREAMBLE
APPLICABILITY AND DEFINITIONS
VERBATIM COPYING
COPYING IN QUANTITY
MODIFICATIONS
COMBINING DOCUMENTS
COLLECTIONS OF DOCUMENTS
AGGREGATION WITH INDEPENDENT WORKS
TRANSLATION
TERMINATION
FUTURE REVISIONS OF THIS LICENSE
ADDENDUM: How to use this License for your documents

Chapter 1. Introduction

Table of Contents

To be a good programmer is difficult and noble. The hardest part of making real a collective vision of a software project is dealing with one's coworkers and customers. Writing computer programs is important and takes great intelligence and skill. But it is really child's play compared to everything else that a good programmer must do to make a software system that succeeds for both the customer and myriad colleagues for whom she is partially responsible. In this essay I attempt to summarize as concisely as possible those things that I wish someone had explained to me when I was twenty-one.

This is very subjective and, therefore, this essay is doomed to be personal and somewhat opinionated. I confine myself to problems that a programmer is very likely to have to face in her work. Many of these problems and their solutions are so general to the human condition that I will probably seem preachy. I hope in spite of this that this essay will be useful.

Computer programming is taught in courses. The excellent books: The Pragmatic Programmer [Prag99], Code Complete [CodeC93], Rapid Development [RDev96], and Extreme Programming Explained [XP99] all teach computer programming and the larger issues of being a good programmer. The essays of Paul Graham[PGSite] and Eric Raymond[Hacker] should certainly be read before or along with this article. This essay differs from those excellent works by emphasizing social problems and comprehensively summarizing the entire set of necessary skills as I see them.

In this essay the term boss to refer to whomever gives you projects to do. I use the words business, company, and tribe, synonymously except that business connotes moneymaking, company connotes the modern workplace and tribe is generally the people you share loyalty with.

Welcome to the tribe.
Chapter 2. Beginner

Table of Contents

Personal Skills

Learn to Debug
How to Debug by Splitting the Problem Space
How to Remove an Error
How to Debug Using a Log
How to Understand Performance Problems
How to Fix Performance Problems
How to Optimize Loops
How to Deal with I/O Expense
How to Manage Memory
How to Deal with Intermittent Bugs
How to Learn Design Skills
How to Conduct Experiments

Team Skills

Why Estimation is Important
How to Estimate Programming Time
How to Find Out Information
How to Utilize People as Information Sources
How to Document Wisely
How to Work with Poor Code
How to Use Source Code Control
How to Unit Test
Take Breaks when Stumped
How to Recognize When to Go Home
How to Deal with Difficult People

Personal Skills
Learn to Debug

Debugging is the cornerstone of being a programmer. The first meaning of the verb to debug is to remove errors, but the meaning that really matters is to see into the execution of a program by examining it. A programmer that cannot debug effectively is blind.

Idealists that think design, or analysis, or complexity theory, or whatnot, are more fundamental are not working programmers. The working programmer does not live in an ideal world. Even if you are perfect, your are surrounded by and must interact with code written by major software companies, organizations like GNU, and your colleagues. Most of this code is imperfect and imperfectly documented. Without the ability to gain visibility into the execution of this code the slightest bump will throw you permanently. Often this visibility can only be gained by experimentation, that is, debugging.

Debugging is about the running of programs, not programs themselves. If you buy something from a major software company, you usually don't get to see the program. But there will still arise places where the code does not conform to the documentation (crashing your entire machine is a common and spectacular example), or where the documentation is mute. More commonly, you create an error, examine the code you wrote and have no clue how the error can be occurring. Inevitably, this means some assumption you are making is not quite correct, or some condition arises that you did not anticipate. Sometimes the magic trick of staring into the source code works. When it doesn't, you must debug.

To get visibility into the execution of a program you must be able to execute the code and observe something about it. Sometimes this is visible, like what is being displayed on a screen, or the delay between two events. In many other cases, it involves things that are not meant to be visible, like the state of some variables inside the code, which lines of code are actually being executed, or whether certain assertions hold across a complicated data structure. These hidden things must be revealed.

The common ways of looking into the ‘innards’ of an executing program can be categorized as:

*

Using a debugging tool,
*

Printlining --- Making a temporary modification to the program, typically adding lines that print information out, and
*

Logging --- Creating a permanent window into the programs execution in the form of a log.

Debugging tools are wonderful when they are stable and available, but the printlining and logging are even more important. Debugging tools often lag behind language development, so at any point in time they may not be available. In addition, because the debugging tool may subtly change the way the program executes it may not always be practical. Finally, there are some kinds of debugging, such as checking an assertion against a large data structure, that require writing code and changing the execution of the program. It is good to know how to use debugging tools when they are stable, but it is critical to be able to employ the other two methods.

Some beginners fear debugging when it requires modifying code. This is understandable---it is a little like exploratory surgery. But you have to learn to poke at the code and make it jump; you have to learn to experiment on it, and understand that nothing that you temporarily do to it will make it worse. If you feel this fear, seek out a mentor---we lose a lot of good programmers at the delicate onset of their learning to this fear.
How to Debug by Splitting the Problem Space

Debugging is fun, because it begins with a mystery. You think it should do something, but instead it does something else. It is not always quite so simple---any examples I can give will be contrived compared to what sometimes happens in practice. Debugging requires creativity and ingenuity. If there is a single key to debugging is to use the divide and conquer technique on the mystery.

Suppose, for example, you created a program that should do ten things in a sequence. When you run it, it crashes. Since you didn't program it to crash, you now have a mystery. When out look at the output, you see that the first seven things in the sequence were run successfully. The last three are not visible from the output, so now your mystery is smaller: ‘It crashed on thing #8, #9, or #10.’

Can you design an experiment to see which thing it crashed on? Sure. You can use a debugger or we can add printline statements (or the equivalent in whatever language you are working in) after #8 and #9. When we run it again, our mystery will be smaller, such as ‘It crashed on thing #9.’ I find that bearing in mind exactly what the mystery is at any point in time helps keep one focused. When several people are working together under pressure on a problem it is easy to forget what the most important mystery is.

The key to divide and conquer as a debugging technique is the same as it is for algorithm design: as long as you do a good job splitting the mystery in the middle, you won't have to split it too many times, and you will be debugging quickly. But what is the middle of a mystery? There is where true creativity and experience comes in.

To a true beginner, the space of all possible errors looks like every line in the source code. You don't have the vision you will later develop to see the other dimensions of the program, such as the space of executed lines, the data structure, the memory management, the interaction with foreign code, the code that is risky, and the code that is simple. For the experience programmer, these other dimensions form an imperfect but very useful mental model of all the things that can go wrong. Having that mental model is what helps one find the middle of the mystery effectively.

Once you have evenly subdivided the space of all that can go wrong, you must try to decide in which space the error lies. In the simple case where the mystery is: ‘Which single unknown line makes my program crash?’, you can ask yourself: ‘Is the unknown line executed before or after this line that I judge to be executed in the about the middle of the running program?’ Usually you will not be so lucky as to know that the error exists in a single line, or even a single block. Often the mystery will be more like: ‘Either there is a pointer in that graph that points to the wrong node, or my algorithm that adds up the variables in that graph doesn't work.’ In that case you may have to write a small program to check that the pointers in the graph are all correct in order to decide which part of the subdivided mystery can be eliminated.
How to Remove an Error

I've intentionally separated the act of examining a program's execution from the act of fixing an error. But of course, debugging does also mean removing the bug. Ideally you will have perfect understanding of the code and will reach an ‘A-Ha!’ moment where you perfectly see the error and how to fix it. But since your program will often use insufficiently documented systems into which you have no visibility, this is not always possible. In other cases the code is so complicated that your understanding cannot be perfect.

In fixing a bug, you want to make the smallest change that fixes the bug. You may see other things that need improvement; but don't fix those at the same time. Attempt to employ the scientific method of changing one thing and only one thing at a time. The best process for this is to be able to easily reproduce the bug, then put your fix in place, and then rerun the program and observe that the bug no longer exists. Of course, sometimes more than one line must be changed, but you should still conceptually apply a single atomic change to fix the bug.

Sometimes, there are really several bugs that look like one. It is up to you to define the bugs and fix them one at a time. Sometimes it is unclear what the program should do or what the original author intended. In this case, you must exercise your experience and judgment and assign your own meaning to the code. Decide what it should do, and comment it or clarify it in some way and then make the code conform to your meaning. This is an intermediate or advanced skill that is sometimes harder than writing the original function in the first place, but the real world is often messy. You may have to fix a system you cannot rewrite.
How to Debug Using a Log

Logging is the practice of writing a system so that it produces a sequence of informative records, called a log. Printlining is just producing a simple, usually temporary, log. Absolute beginners must understand and use logs because their knowledge of the programming is limited; system architects must understand and use logs because of the complexity of the system. The amount of information that is provided by the log should be configurable, ideally while the program is running. In general, logs offer three basic advantages:

*

Logs can provide useful information about bugs that are hard to reproduce (such as those that occur in the production environment but that cannot be reproduced in the test environment).
*

Logs can provide statistics and data relevant to performance, such as the time passing between statements.
*

When configurable, logs allow general information to be captured in order to debug unanticipated specific problems without having to modify and/or redeploy the code just to deal with those specific problems.

The amount to output into the log is always a compromise between information and brevity. Too much information makes the log expensive and produces scroll blindness, making it hard to find the information you need. Too little information and it may not contain what you need. For this reason, making what is output configurable is very useful. Typically, each record in the log will identify its position in the source code, the thread that executed it if applicable, the precise time of execution, and, commonly, an additional useful piece of information, such as the value of some variable, the amount of free memory, the number of data objects, etc. These log statements are sprinkled throughout the source code but are particularly at major functionality points and around risky code. Each statement can be assigned a level and will only output a record if the system is currently configured to output that level. You should design the log statements to address problems that you anticipate. Anticipate the need to measure performance.

If you have a permanent log, printlining can now be done in terms of the log records, and some of the debugging statements will probably be permanently added to the logging system.
How to Understand Performance Problems

Learning to understand the performance of a running system is unavoidable for the same reason that learning debugging is. Even if the code you understand perfectly precisely the cost of the code you write, your code will make calls into other software systems that you have little control over or visibility into. However, in practice performance problems are a little different and a little easier than debugging in general.

Suppose that you or your customers consider a system or a subsystem to be too slow. Before you try to make it faster, you must build a mental model of why it is slow. To do this you can use a profiling tool or a good log to figure out where the time or other resources are really being spent. There is a famous dictum that 90% of the time will be spent in 10% of the code. I would add to that the importance of input/output expense (I/O) to performance issues. Often most of the time is spent in I/O in one way or another. Finding the expensive I/O and the expensive 10% of the code is a good first step to building your mental model.

There are many dimensions to the performance of a computer system, and many resources consumed. The first resource to measure is wall--clock time, the total time that passes for the computation. Logging wall-clock time is particularly valuable because it can inform about unpredictable circumstance that arise in situations where other profiling is impractical. However, this may not always represent the whole picture. Sometimes something that takes a little longer but doesn't burn up so many processor seconds will be much better in computing environment you actually have to deal with. Similarly, memory, network bandwidth, database or other server accesses may, in the end, be far more expensive than processor seconds.

Contention for shared resources that are synchronized can cause deadlock and starvation. Deadlock is the inability to proceed because of improper synchronization or resource demands. Starvation is the failure to schedule a component properly. If it can be at all anticipated, it is best to have a way of measuring this contention from the start of your project. Even if this contention does not occur, it is very helpful to be able to assert that with confidence.
How to Fix Performance Problems

Most software projects can be made with relatively little effort 10 to 100 times faster than they are at the they are first released. Under time-to-market pressure, it is both wise and effective to choose a solution that gets the job done simply and quickly, but less efficiently than some other solution. However, performance is a part of usability, and often it must eventually be considered more carefully.

The key to improving the performance of a very complicated system is to analyze it well enough to find the bottlenecks, or places where most of the resources are consumed. There is not much sense in optimizing a function that accounts for only 1% of the computation time. As a rule of thumb you should think carefully before doing anything unless you think it is going to make the system or a significant part of it at least twice as fast. There is usually a way to do this. Consider the test and quality assurance effort that your change will require. Each change brings a test burden with it, so it is much better to have a few big changes.

After you've made a two-fold improvement in something, you need to at least rethink and perhaps reanalyze to discover the next-most-expensive bottleneck in the system, and attack that to get another two-fold improvement.

Often, the bottlenecks in performance will be an example of counting cows by counting legs and dividing by four, instead of counting heads. For example, I've made errors such as failing to provide a relational database system with a proper index on a column I look up a lot, which probably made it at least 20 times slower. Other examples include doing unnecessary I/O in inner loops, leaving in debugging statements that are no longer needed, unnecessary memory allocation, and, in particular, inexpert use of libraries and other subsystems that are often poorly documented with respect to performance. This kind of improvement is sometimes called low-hanging fruit, meaning that it can be easily picked to provide some benefit.

What do you do when you start to run out of low-hanging fruit? Well, you can reach higher, or chop the tree down. You can continue making small improvements or you can seriously redesign a system or a subsystem. (This is a great opportunity to use your skills as a good programmer, not only in the new design but also in convincing your boss that this is a good idea.) However, before you argue for the redesign of a subsystem, you should ask yourself whether or not your proposal will make it five to ten time better.
How to Optimize Loops

Sometimes you'll encounter loops, or recursive functions, that take a long time to execute and are bottlenecks in your product. Before you try to make the loop a little faster, but spend a few minutes considering if there is a way to remove it entirely. Would a different algorithm do? Could you compute that while computing something else? If you can't find away around it, then you can optimize the loop. This is simple; move stuff out. In the end, this will require not only ingenuity but also an understanding of the expense of each kind of statement and expression. Here are some suggestions:

*

Remove floating point operations.
*

Don't allocate new memory blocks unnecessarily.
*

Fold constants together.
*

Move I/O into a buffer.
*

Try not to divide.
*

Try not to do expensive typecasts.
*

Move a pointer rather than recomputing indices.

The cost of each of these operations depends on your specific system. On some systems compilers and hardware do these things for you. Clear, efficient code is better than code that requires an understanding of a particular platform.
How to Deal with I/O Expense

For a lot of problems, processors are fast compared to the cost of communicating with a hardware device. This cost is usually abbreviated I/O, and can include network cost, disk I/O, database queries, file I/O, and other use of some hardware not very close to the processor. Therefore building a fast system is often more a question of improving I/O than improving the code in some tight loop, or even improving an algorithm.

There are two very fundamental techniques to improving I/O: caching and representation. Caching is avoiding I/O (generally avoiding the reading of some abstract value) by storing a copy of that value locally so no I/O is performed to get the value. The first key to caching is to make it crystal clear which data is the master and which are copies. There is only one master---period. Caching brings with it the danger that the copy is sometimes can't reflect changes to the master instantaneously.

Representation is the approach of making I/O cheaper by representing data more efficiently. This is often in tension with other demands, like human readability and portability.

Representations can often be improved by a factor of two or three from their first implementation. Techniques for doing this include using a binary representation instead of one that is human readable, transmitting a dictionary of symbols along with the data so that long symbols don't have to be encoded, and, at the extreme, things like Huffman encoding.

A third technique that is sometimes possible is to improve the locality of reference by pushing the computation closer to the data. For instance, if you are reading some data from a database and computing something simple from it, such as a summation, try to get the database server to do it for you. This is highly dependent on the kind of system you're working with, but you should explore it.
How to Manage Memory

Memory is a precious resource that you can't afford to run out of. You can ignore it for a while but eventually you will have to decide how to manage memory.

Space that needs to persist beyond the scope of a single subroutine is often called heap allocated. A chunk of memory is useless, hence garbage, when nothing refers to it. Depending on the system you use, you may have to explicitly deallocate memory yourself when it is about to become garbage. More often you may be able to use a system that provides a garbage collector. A garbage collector notices garbage and frees its space without any action required by the programmer. Garbage collection is wonderful: it lessens errors and increases code brevity and concision cheaply. Use it when you can.

But even with garbage collection, you can fill up all memory with garbage. A classic mistake is to use a hash table as a cache and forget to remove the references in the hash table. Since the reference remains, the referent is noncollectable but useless. This is called a memory leak. You should look for and fix memory leaks early. If you have long running systems memory may never be exhausted in testing but will be exhausted by the user.

The creation of new objects is moderately expensive on any system. Memory allocated directly in the local variables of a subroutine, however, is usually cheap because the policy for freeing it can be very simple. You should avoid unnecessary object creation.

An important case occurs when you can define an upper bound on the number of objects you will need at one time. If these objects all take up the same amount of memory, you may be able to allocate a single block of memory, or a buffer, to hold them all. The objects you need can be allocated and released inside this buffer in a set rotation pattern, so it is sometimes called a ring buffer. This is usually faster than heap allocation.

Sometimes you have to explicitly free allocated space so it can be reallocated rather than rely on garbage collection. Then you must apply careful intelligence to each chunk of allocated memory and design a way for it to be deallocated at the appropriate time. The method may differ for each kind of object you create. You must make sure that every execution of a memory allocating operation is matched by a memory deallocating operation eventually. This is so difficult that programmers often simply implement a rudimentary form or garbage collection, such as reference counting, to do this for them.
How to Deal with Intermittent Bugs

The intermittent bug is a cousin of the 50-foot-invisible-scorpion-from-outer-space kind of bug. This nightmare occurs so rarely that it is hard to observe, yet often enough that it can't be ignored. You can't debug because you can't find it.

Although after 8 hours you will start to doubt it, the intermittent bug has to obey the same laws of logic everything else does. What makes it hard is that it occurs only under unknown conditions. Try to record the circumstances under which the bug does occur, so that you can guess at what the variability really is. The condition may be related to data values, such as ‘This only happens when we enter Wyoming as a value.’ If that is not the source of variability, the next suspect should be improperly synchronized concurrency.

Try, try, try to reproduce the bug in a controlled way. If you can't reproduce it, set a trap for it by building a logging system, a special one if you have to, that can log what you guess you need when it really does occur. Resign yourself to that if the bug only occurs in production and not at your whim, this is may be a long process. The hints that you get from the log may not provide the solution but may give you enough information to improve the logging. The improved logging system may take a long time to be put into production. Then, you have to wait for the bug to reoccur to get more information. This cycle can go on for some time.

The stupidest intermittent bug I ever created was in a multi-threaded implementation of a functional programming language for a class project. I had very carefully insured correct concurrent evaluation of the functional program, good utilization of all the CPUs available (eight, in this case). I simply forgot to synchronize the garbage collector. The system could run a long time, often finishing whatever task I began, before anything noticeable went wrong. I'm ashamed to admit I had begun to question the hardware before my mistake dawned on me.

At work we recently had an intermittent bug that took us several weeks to find. We have multi-threaded application servers in Java™ behind Apache™ web servers. To maintain fast page turns, we do all I/O in small set of four separate threads that are different than the page-turning threads. Every once in a while these would apparently get ‘stuck’ and cease doing anything useful, so far as our logging allowed us to tell, for hours. Since we had four threads, this was not in itself a giant problem---unless all four got stuck. Then the queues emptied by these threads would quickly fill up all available memory and crash our server. It took us about a week to figure this much out, and we still didn't know what caused it, when it would happen, or even what the threads where doing when they got ‘stuck’.

This illustrates some risk associated with third-party software. We were using a licensed piece of code that removed HTML tags from text. Due to its place of origin we affectionately referred to this as ‘the French stripper.’ Although we had the source code (thank goodness!) we had not studied it carefully until by turning up the logging on our servers we finally realized that the email threads were getting stuck in the French stripper.

The stripper performed well except on some long and unusual kinds of texts. On these texts, the code was quadratic or worse. This means that the processing time was proportional to the square of the length of the text. Had these texts occurred commonly, we would have found the bug right away. If they had never occurred at all, we would never have had a problem. As it happens, it took us weeks to finally understand and resolve the problem.
How to Learn Design Skills

To learn how to design software, study the action of a mentor by being physically present when they are designing. Then study well-written pieces of software. After that, you can read some books on the latest design techniques.

Then you must do it yourself. Start with a small project. When you are finally done, consider how the design failed or succeeded and how you diverged from your original conception. They move on to larger projects, hopefully in conjunction with other people. Design is a matter of judgment that takes years to acquire. A smart programmer can learn the basics adequately in two months and can improve from there.

It is natural and helpful to develop your own style, but remember that design is an art, not a science. People who write books on the subject have a vested interest in making it seem scientific. Don't become dogmatic about particular design styles.
How to Conduct Experiments

The late, great Edsger Dijkstra has eloquently explained that Computer Science is not an experimental science[ExpCS] and doesn't depend on electronic computers. As he puts it referring to the 1960s[Knife],

...the harm was done: the topic became known as “computer science”---which, actually, is like referring to surgery as “knife science” --- and it was firmly implanted in people's minds that computing science is about machines and their peripheral equipment.

Programming ought not to be an experimental science, but most working programmers do not have the luxury of engaging in what Dijkstra means by computing science. We must work in the realm of experimentation, just as some, but not all, physicists do. If thirty years from now programming can be performed without experimentation, it will be a great accomplishment of Computer Science.

The kinds of experiments you will have to perform include:

*

Testing systems with small examples to verify that they conform to the documentation or to understand their response when there is no documentation,
*

Testing small code changes to see if they actually fix a bug,
*

Measuring the performance of a system under two different conditions due to imperfect knowledge of there performance characteristics,
*

Checking the integrity of data, and
*

Collecting statistics that may hint at the solution to difficult or hard-to-repeat bugs.

I don't think in this essay I can explain the design of experiments; you will have to study and practice. However, I can offer two bits of advice.

First, try to be very clear about your hypothesis, or the assertion that you are trying to test. It also helps to write the hypothesis down, especially if you find yourself confused or are working with others.

You will often find yourself having to design a series of experiments, each of which is based on the knowledge gained from the last experiment. Therefore, you should design your experiments to provide the most information possible. Unfortunately, this is in tension with keeping each experiment simple---you will have to develop this judgment through experience.
Team Skills
Why Estimation is Important

To get a working software system in active use as quickly as possible requires not only planning the development, but also planning the documentation, deployment, marketing. In a commercial project it also requires sales and finance. Without predictability of the development time, it is impossible to plan these effectively.

Good estimation provides predictability. Managers love it, as well they should. The fact that it is impossible, both theoretically and practically, to predict accurately how long it will take to develop software is often lost on managers. We are asked to do this impossible thing all the time, and we must face up to it honestly. However, it would be dishonest not to admit the impossibility of this task, and when necessary, explain it. There is a lot of room for miscommunication about estimates, as people have a startling tendency to think wishfully that the sentence:

I estimate that, if I really understand the problem, it is about 50% likely that we will be done in five weeks (if no one bothers us during that time).

really means:

I promise to have it all done five weeks from now.

This common interpretation problem requires that you explicitly discuss what the estimate means with your boss or customer as if they were a simpleton. Restate your assumptions, no matter how obvious they seem to you.
How to Estimate Programming Time

Estimation takes practice. It also takes labor. It takes so much labor it may be a good idea to estimate the time it will take to make the estimate, especially if you are asked to estimate something big.

When asked to provide an estimate of something big, the most honest thing to do is to stall. Most engineers are enthusiastic and eager to please, and stalling certainly will displease the stalled. But an on-the-spot estimate probably won't be accurate and honest.

While stalling, it may be possible to consider doing or prototyping the task. If political pressure permits, this is the most accurate way of producing the estimate, and it makes real progress.

When not possible to take the time for some investigation, you should first establish the meaning of the estimate very clearly. Restate that meaning as the first and last part of your written estimate. Prepare a written estimate by deconstructing the task into progressively smaller subtasks until each small task is no more than a day; ideally at most in length. The most important thing is not to leave anything out. For instance, documentation, testing, time for planning, time for communicating with other groups, and vacation time are all very important. If you spend part of each day dealing with knuckleheads, put a line item for that in the estimate. This gives your boss visibility into what is using up your time at a minimum, and might get you more time.

I know good engineers who pad estimates implicitly, but I recommend that you do not. One of the results of padding is trust in you may be depleted. For instance, an engineer might estimate three days for a task that she truly thinks will take one day. The engineer may plan to spend two days documenting it, or two days working on some other useful project. But it will be detectable that the task was done in only one day (if it turns out that way), and the appearance of slacking or overestimating is born. It's far better to give proper visibility into what you are actually doing. If documentation takes twice as long as coding and the estimate says so, tremendous advantage is gained by making this visible to the manager.

Pad explicitly instead. If a task will probably take one day---but might take ten days if your approach doesn't work---note this somehow in the estimate if you can; if not, at least do an average weighted by your estimates of the probabilities. Any risk factor that you can identify and assign an estimate to should go into the schedule. One person is unlikely to be sick in any given week. But a large project with many engineers will have some sick time; likewise vacation time. And what is the probability of a mandatory company-wide training seminar? If it can be estimated, stick it in. There are of course, unknown unknowns, or unk-unks. Unk-unks by definition cannot be estimated individually. You can try to create a global line item for all unk-unks, or handle them in some other way that you communicate to your boss. You cannot, however, let your boss forget that they exist, and it is devilishly easy for an estimate to become a schedule without the unk-unks considered.

In a team environment, you should try to have the people who will do the work do the estimate, and you should try to have team-wide consensus on estimates. People vary widely in skill, experience, preparedness, and confidence. Calamity strikes when a strong programmer estimates for herself and then weak programmers are held to this estimate. The act of having the whole team agree on a line-by-line basis to the estimate clarifies the team understanding, as well as allowing the opportunity for tactical reassignment of resources (for instance, shifting burden away from weaker team members to stronger).

If there are big risks that cannot be evaluated, it is your duty to state so forcefully enough that your manager does not commit to them and then become embarrassed when the risk occurs. Hopefully in such a case whatever is needed will be done to decrease the risk.

If you can convince your company to use Extreme Programming, you will only have to estimate relatively small things, and this is both more fun and more productive.
How to Find Out Information

The nature of what you need to know determines how you should find it.

If you need information about concrete things that are objective and easy to verify, for example the latest patch level of a software product, ask a large number of people politely by searching the internet for it or by posting on a discussion group. Don't search on the internet for anything that smacks of either opinion or subjective interpretation: the ratio of drivel to truth is too high.

If you need general knowledge about something subjective the history of what people have thought about it, go to the library (the physical building in which books are stored). For example, to learn about math or mushrooms or mysticism, go to the library.

If you need to know how to do something that is not trivial get two or three books on the subject and read them. You might learn how to do something trivial, like install a software package, from the Internet. You can even learn important things, like good programming technique, but you can easily spend more time searching and sorting the results and attempting to divine the authority of the results than it would take to read the pertinent part of a solid book.

If you need information that no one else could be expected to know for example, ‘does this software that is brand new work on gigantic data sets?’, you must still search the internet and the library. After those options are completely exhausted, you may design an experiment to ascertain it.

If you want an opinion or a value judgment that takes into account some unique circumstance, talk to an expert. For instance, if you want to know whether or not it is a good idea to build a modern database management system in LISP, you should talk to a LISP expert and a database expert.

If you want to know how likely it is that a faster algorithm for a particular application exists that has not yet been published, talk to someone working in that field.

If you want to make a personal decision that only you can make like whether or not you should start a business, try putting into writing a list of arguments for and against the idea. If that fails, consider divination. Suppose you have studied the idea from all angles, have done all your homework, and worked out all the consequences and pros and cons in your mind, and yet still remain indecisive. You now must follow your heart and tell your brain to shut up. The multitude of available divination techniques are very useful for determining your own semi-conscious desires, as they each present a complete ambiguous and random pattern that your own subconscious will assign meaning to.
How to Utilize People as Information Sources

Respect every person's time and balance it against your own. Asking someone a question accomplishes far more than just receiving the answer. The person learns about you, both by enjoying your presence and hearing the particular question. You learn about the person in the same way, and you may learn the answer you seek. This is usually far more important than your question.

However, the value of this diminishes the more you do it. You are, after all, using the most precious commodity a person has: their time. The benefits of communication must be weighed against the costs. Furthermore, the particular costs and benefits derived differ from person to person. I strongly believe that an executive of 100 people should spend five minutes a month talking to each person in her organization, which would be about 5% of their time. But ten minutes might be too much, and five minutes is too much if they have one thousand employees. The amount of time you spend talking to each person in your organization depends on their role (more than their position). You should talk to your boss more than your boss's boss, but you should talk to your boss's boss a little. It may be uncomfortable, but I believe you have a duty to talk a little bit to all your superiors, each month, no matter what.

The basic rule is that everyone benefits from talking to you a little bit, and the more they talk to you, the less benefit they derive. It is your job to provide them this benefit, and to get the benefit of communicating with them, keeping the benefit in balance with the time spent.

It is important to respect your own time. If talking to someone, even if it will cost them time, will save you a great deal of time, then you should do it unless you think their time is more valuable than yours, to the tribe, by that factor.

A strange example of this is the summer intern. A summer intern in a highly technical position can't be expected to accomplish too much; they can be expected to pester the hell out of everybody there. So why is this tolerated? Because the pestered are receiving something important from the intern. They get a chance to showoff a little. They get a chance to hear some new ideas, maybe; they get a chance to see things from a different perspective. They may also be trying to recruit the intern, but even if this is not the case there is much to gain.

You should ask people for a little bit of their wisdom and judgment whenever you honestly believe they have something to say. This flatters them and you will learn something and teach them something. A good programmer does not often need the advice of a Vice President of Sales, but if you ever do, you be sure to ask for it. I once asked to listen in on a few sales calls to better understand the job of our sales staff. This took no more than 30 minutes but I think that small effort made an impression on the sales force.
How to Document Wisely

Life is too short to write crap nobody will read; if you write crap, nobody will read it. Therefore a little good documentation is best. Managers often don't understand this, because even bad documentation gives them a false sense of security that they are not dependent on their programmers. If someone absolutely insists that you write truly useless documentation, say ``yes'' and quietly begin looking for a better job.

There's nothing quite as effective as putting an accurate estimate of the amount of time it will take to produce good documentation into an estimate to slacken the demand for documentation. The truth is cold and hard: documentation, like testing, can take many times longer than developing code.

Writing good documentation is, first of all, good writing. I suggest you find books on writing, study them, and practice. But even if you are a lousy writer or have poor command of the language in which you must document, the Golden Rule is all you really need: ``Do unto others as you would have them do unto you.'' Take time to really think about who will be reading your documentation, what they need to get out of it, and how you can teach that to them. If you do that, you will be an above average documentation writer, and a good programmer.

When it comes to actually documenting code itself, as opposed to producing documents that can actually be read by non-programmers, the best programmers I've ever known hold a universal sentiment: write self-explanatory code and only document code in the places that you cannot make it clear by writing the code itself. There are two good reasons for this. First, anyone who needs to see code-level documentation will in most cases be able to and prefer to read the code anyway. Admittedly, this seems easier to the experienced programmer than to the beginner. More importantly however, is that the code and the documentation cannot be inconsistent if there is no documentation. The source code can at worst be wrong and confusing. The documentation, if not written perfectly, can lie, and that is a thousand times worse.

This does not make it easier on the responsible programmer. How does one write self-explanatory code? What does that even mean? It means:

*

Writing code knowing that someone will have to read it;
*

Applying the golden rule;
*

Choosing a solution that is straightforward, even if you could get by with another solution faster;
*

Sacrificing small optimizations that obfuscate the code;
*

Thinking about the reader and spending some of your precious time to make it easier on her; and
*

Not ever using a function name like ``foo'',``bar'', or ``doIt''!

How to Work with Poor Code

It is very common to have to work with poor quality code that someone else has written. Don't think too poorly of them, however, until you have walked in their shoes. They may have been asked very consciously to get something done quickly to meet schedule pressure. Regardless, in order to work with unclear code you must understand it. To understand it takes learning time, and that time will have to come out of some schedule, somewhere, and you must insist on it. To understand it, you will have to read the source code. You will probably have to experiment with it.

This is a good time to document, even if it is only for yourself, because the act of trying to document the code will force you to consider angles you might not have considered, and the resulting document may be useful. While you're doing this, consider what it would take to rewrite some or all of the code. Would it actually save time to rewrite some of it? Could you trust it better if you rewrote it? Be careful of arrogance here. If you rewrite it, it will be easier for you to deal with, but will it really be easier for the next person who has to read it? If you rewrite it, what will the test burden be? Will the need to re-test it outweigh any benefits that might be gained?

In any estimate that you make for work against code you didn't write, the quality of that code should affect your perception of the risk of problems and unk-unks.

It is important to remember that abstraction and encapsulation, two of a programmer's best tools, are particularly applicable to lousy code. You may not be able to redesign a large block of code, but if you can add a certain amount of abstraction to it you can obtain some of the benefits of a good design without reworking the whole mess. In particular, you can try to wall off the parts that are particularly bad so that they may be redesigned independently.
How to Use Source Code Control

Source code control systems let you manage projects effectively. They're very useful for one person and essential for a group. They track all changes in different versions so that no code is ever lost and meaning can be assigned to changes. One can create throw-away and debugging code with confidence with a source code control system, since the code you modify is kept carefully separate from committed, official code that will be shared with the team or released.

I was late to appreciate the benefits of source code control systems but now I wouldn't live without one even on a one-person project. Generally they are necessary when you have team working on the same code base. However, they have another great advantage: they encourage thinking about the code as a growing, organic system. Since each change is marked as a new revision with a new name or number, one begins to think of the software as a visibly progressive series of improvements. I think this is especially useful for beginners.

A good technique for using a source code control system is to stay within a few days of being up-to-date at all time. Code that can't be finished in a few days is checked in, but in a way that it is inactive and will not be called, and therefore not create any problems for anybody else. Committing a mistake that slows down your teammates is a serious error; it is often taboo.
How to Unit Test

Unit testing, the testing of an individual piece of coded functionality by the team that wrote it, is a part of coding, not something different from it. Part of designing the code is designing how it will be tested. You should write down a test plan, even if it is only one sentence. Sometimes the test will be simple: ``Does the button look good?'' Sometimes it will be complex: ``Did this matching algorithm return precisely the correct matches?''

Use assertion checking and test drivers whenever possible. This not only catches bugs early, but is very useful later on and lets you eliminate mysteries that you would otherwise have to worry about.

The Extreme Programming developers are writing extensively on unit testing effectively; I can do no better than to recommend their writings.
Take Breaks when Stumped

When stumped, take a break. I sometimes meditate for 15 minutes when stumped and the problem magically unravels when I come back to it. A night's sleep sometimes does the same thing on a larger scale. It's possible that temporarily switching to any other activity may work.
How to Recognize When to Go Home

Computer programming is an activity that is also a culture. The unfortunate fact is that it is not a culture that values mental or physical health very much. For both cultural/historical reasons (the need to work at night on unloaded computers, for example) and because of overwhelming time-to-market pressure and the scarcity of programmers, computer programmers are traditionally overworked. I don't think you can trust all the stories you hear, but I think 60 hours a week is common, and 50 is pretty much a minimum. This means that often much more than that is required. This is serious problem for a good programmer, who is responsible not only for themselves but their teammates as well. You have to recognize when to go home, and sometimes when to suggest that other people go home. There can't be any fixed rules for solving this problem, anymore than there can be fixed rules for raising a child, for the same reason---every human being is different.

Beyond 60 hours a week is an extraordinary effort for me, which I can apply for short periods of time (about one week), and that is sometimes expected of me. I don't know if it is fair to expect 60 hours of work from a person; I don't even know if 40 is fair. I am sure, however, that it is stupid to work so much that you are getting little out of that extra hour you work. For me personally, that's any more than 60 hours a week. I personally think a programmer should exercise noblesse oblige and shoulder a heavy burden. However, it is not a programmer's duty to be a patsy. The sad fact is programmers are often asked to be patsies in order to put on a show for somebody, for example a manager trying to impress an executive. Programmers often succumb to this because they are eager to please and not very good at saying no. There are four defenses against this:

*

Communicate as much as possible with everyone in the company so that no one can mislead the executives about what is going on,
*

Learn to estimate and schedule defensively and explicitly and give everyone visibility into what the schedule is and where it stands,
*

Learn to say no, and say no as a team when necessary, and
*

Quit if you have to.

Most programmers are good programmers, and good programmers want to get a lot done. To do that, they have to manage their time effectively. There is a certain amount of mental inertia associated with getting warmed-up to a problem and deeply involved in it. Many programmers find they work best when they have long, uninterrupted blocks of time in which to get warmed-up and concentrate. However, people must sleep and perform other duties. Each person needs to find a way to satisfy both their human rhythm and their work rhythm. Each programmer needs to do whatever it takes to procure efficient work periods, such as reserving certain days in which you will attend only the most critical meetings.

Since I have children, I try to spend evenings with them sometimes. The rhythm that works best for me is to work a very long day, sleep in the office or near the office (I have a long commute from home to work) then go home early enough the next day to spend time with my children before they go to bed. I am not comfortable with this, but it is the best compromise I have been able to work out. Go home if you have a contagious disease. You should go home if you are thinking suicidal thoughts. You should take a break or go home if you think homicidal thoughts for more than a few seconds. You should send someone home if they show serious mental malfunctioning or signs of mental illness beyond mild depression. If you are tempted to be dishonest or deceptive in a way that you normally are not due to fatigue, you should take a break. Don't use cocaine or amphetamines to combat fatigue. Don't abuse caffeine.
How to Deal with Difficult People

You will probably have to deal with difficult people. You may even be a difficult person yourself. If you are the kind of person who has a lot of conflicts with coworkers and authority figures, you should cherish the independence this implies, but work on your interpersonal skills without sacrificing your intelligence or principles.

This can be very disturbing to some programmers who have no experience in this sort of thing and whose previous life experience has taught them patterns of behavior that are not useful in the workplace. Difficult people are often inured to disagreement and they are less affected by social pressure to compromise than others. The key is to respect them appropriately, which is more than you will want to but not as much as they might want.

Programmers have to work together as a team. When disagreement arises, it must be resolved somehow, it cannot be ducked for long. Difficult people are often extremely intelligent and have something very useful to say. It is critical that you listen and understand the difficult person without prejudice caused by the person. A failure to communicate is often the basis of disagreement but it can sometimes be removed with great patience. Try to keep this communication cool and cordial, and don't accept any baits for greater conflict that may be offered. After a reasonable period of trying to understand, make a decision.

Don't let a bully force you to do something you don't agree with. If you are the leader, do what you think is best. Don't make a decision for any personal reasons, and be prepared to explain the reasons for your decision. If you are a teammate with a difficult person, don't let the leader's decision have any personal impact. If it doesn't go your way, do it the other way whole-heartedly.

Difficult people do change and improve. I've seen it with my own eyes, but it is very rare. However, everyone has transitory ups and downs.

One of the challenges that every programmer but especially leaders face is keeping the difficult person fully engaged. They are more prone to duck work and resist passively than others.
Chapter 3. Intermediate

Table of Contents

Personal Skills

How to Stay Motivated
How to be Widely Trusted
How to Tradeoff Time vs. Space
How to Stress Test
How to Balance Brevity and Abstraction
How to Learn New Skills
Learn to Type
How to Do Integration Testing
Communication Languages
Heavy Tools
How to analyze data

Team Skills

How to Manage Development Time
How to Manage Third-Party Software Risks
How to Manage Consultants
How to Communicate the Right Amount
How to Disagree Honestly and Get Away with It

Judgement

How to Tradeoff Quality Against Development Time
How to Manage Software System Dependence
How to Decide if Software is Too Immature
How to Make a Buy vs. Build Decision
How to Grow Professionally
How to Evaluate Interviewees
How to Know When to Apply Fancy Computer Science
How to Talk to Non-Engineers

Personal Skills
How to Stay Motivated

It is a wonderful and surprising fact that programmers are highly motivated by the desire to create artifacts that are beautiful, useful, or nifty. This desire is not unique to programmers nor universal but it is so strong and common among programmers that it separates them from others in other roles.

This has practical and important consequences. If programmers are asked to do something that is not beautiful, useful, or nifty, they will have low morale. There's a lot of money to be made doing ugly, stupid, and boring stuff; but in the end, fun will make the most money for the company.

Obviously, there are entire industries organized around motivational techniques some of which apply here. The things that are specific to programming that I can identify are:

*

Use the best language for the job.
*

Look for opportunities to apply new techniques, languages, and technologies.
*

Try to either learn or teach something, however small, in each project.

Finally, if possible, measure the impact of your work in terms of something that will be personally motivating. For example, when fixing bugs, counting the number of bugs that I have fixed is not at all motivational to me, because it is independent of the number that may still exist, and is also affects the total value I'm adding to my company's customers in only the smallest possible way. Relating each bug to a happy customer, however, is personally motivating to me.
How to be Widely Trusted

To be trusted you must be trustworthy. You must also be visible. If know one knows about you, no trust will be invested in you. With those close to you, such as your teammates, this should not be an issue. You establish trust by being responsive and informative to those outside your department or team. Occasionally someone will abuse this trust, and ask for unreasonable favors. Don't be afraid of this, just explain what you would have to give up doing to perform the favor.

Don't pretend to know something that you don't. With people that are not teammates, you may have to make a clear distinction between ``not knowing right off the top of my head'' and ``not being able to figure it out, ever.''
How to Tradeoff Time vs. Space

You can be a good programmer without going to college, but you can't be a good intermediate programmer without knowing basic computational complexity theory. You don't need to know ``big O'' notation, but I personally think you should be able to understand the difference between ``constant-time'',``n log n'' and ``n squared''. You might be able to intuit how to tradeoff time against space without this knowledge, but in its absence you will not have a firm basis for communicating with your colleagues.

In designing or understanding an algorithm, the amount of time it takes to run is sometimes a function of the size of the input. When that is true, we can say an algorithm's worst/expected/best-case running time is ``n log n'' if it is proportional to the size ($n$) times the logarithm of the size. The notation and way of speaking can be also be applied to the space taken up by a data structure.

To me, computational complexity theory is beautiful and as profound as physics---and a little bit goes a long way!

Time (processor cycles) and space (memory) can be traded off against each other. Engineering is about compromise, and this is a fine example. It is not always systematic. In general, however, one can save space by encoding things more tightly, at the expense of more computation time when you have to decode them. You can save time by caching, that is, spending space to store a local copy of something, at the expense of having to maintain the consistency of the cache. You can sometimes save time by maintaining more information in a data structure. This usually cost a small amount of space but may complicate the algorithm.

Improving the space/time tradeoff can often change one or the other dramatically. However, before you work on this you should ask yourself if what you are improving is really the thing that needs the most improvement. It's fun to work on an algorithm, but you can't let that blind you to the cold hard fact that improving something that is not a problem will not make any noticeable difference and will create a test burden.

Memory on modern computers appears cheap, because unlike processor time, you can't see it being used until you hit the wall; but then failure is catastrophic. There are also other hidden costs to using memory, such as your effect on other programs that must be resident, and the time to allocate and deallocate it. Consider this carefully before you trade away space to gain speed.
How to Stress Test

Stress testing is fun. At first it appears that the purpose of stress testing is to find out if the system works under a load. In reality, it is common that the system does work under a load but fails to work in some way when the load is heavy enough. I call this hitting the wall or bonking[1]. There may be some exceptions, but there is almost always a ‘wall’. The purpose of stress testing is to figure out where the wall is, and then figure out how to move the wall further out.

A plan for stress testing should be developed early in the project, because it often helps to clarify exactly what is expected. Is two seconds for a web page request a miserable failure or a smashing success? Is 500 concurrent users enough? That, of course, depends, but one must know the answer when designing the system that answers the request. The stress test needs to model reality well enough to be useful. It isn't really possible to simulate 500 erratic and unpredictable humans using a system concurrently very easily, but one can at least create 500 simulations and try to model some part of what they might do.

In stress testing, start out with a light load and load the system along some dimension---such as input rate or input size---until you hit the wall. If the wall is too close to satisfy your needs, figure out which resource is the bottleneck (there is usually a dominant one.) Is it memory, processor, I/O, network bandwidth, or data contention? Then figure out how you can move the wall. Note that moving the wall, that is, increasing the maximum load the system can handle, might not help or might actually hurt the performance of a lightly loaded system. Usually performance under heavy load is more important than performance under a light load.

You may have to get visibility into several different dimensions to build up a mental model of it; no single technique is sufficient. For instance, logging often gives a good idea of the wall-clock time between two events in the system, but unless carefully constructed, doesn't give visibility into memory utilization or even data structure size. Similarly, in a modern system, a number of computers and many software systems may be cooperating. Particularly when you are hitting the wall (that is, the performance is non-linear in the size of the input) these other software systems may be a bottleneck. Visibility into these systems, even if only measuring the processor load on all participating machines, can be very helpful.

Knowing where the wall is is essential not only to moving the wall, but also to providing predictability so that the business can be managed effectively.
How to Balance Brevity and Abstraction

Abstraction is key to programming. You should carefully choose how abstract you need to be. Beginning programmers in their enthusiasm often create more abstraction than is really useful. One sign of this is if you create classes that don't really contain any code and don't really do anything except serve to abstract something. The attraction of this is understandable but the value of code brevity must be measured against the value of abstraction. Occasionally, one sees a mistake made by enthusiastic idealists: at the start of the project a lot of classes are defined that seem wonderfully abstract and one may speculate that they will handle every eventuality that may arise. As the project progresses and fatigue sets in, the code itself becomes messy. Function bodies become longer than they should be. The empty classes are a burden to document that is ignored when under pressure. The final result would have been better if the energy spent on abstraction had been spent on keeping things short and simple. This is a form of speculative programming. I strongly recommend the article ``Succinctness is Power'' by Paul Graham[PGSite].

There is a certain dogma associated with useful techniques such as information hiding and object oriented programming that are sometimes taken too far. These techniques let one code abstractly and anticipate change. I personally think, however, that you should not produce much speculative code. For example, it is an accepted style to hide an integer variable on an object behind mutators and accessors, so that the variable itself is not exposed, only the little interface to it. This does allow the implementation of that variable to be changed without affecting the calling code, and is perhaps appropriate to a library writer who must publish a very stable API. But I don't think the benefit of this outweighs the cost of the wordiness of it when my team owns the calling code and hence can recode the caller as easily as the called. Four or five extra lines of code is a heavy price to pay for this speculative benefit.

Portability poses a similar problem. Should code be portable to a different computer, compiler, software system or platform, or simply easily ported? I think a non-portable, short-and-easily-ported piece of code is better than a long portable one. It is relatively easy and certainly a good idea to confine non-portable code to designated areas, such as a class that makes database queries that are specific to a given DBMS.
How to Learn New Skills

Learning new skills, especially non-technical ones, is the greatest fun of all. Most companies would have better morale if they understood how much this motivates programmers.

Humans learn by doing. Book-reading and class-taking are useful. But could you have any respect for a programmer who had never written a program? To learn any skill, you have to put yourself in a forgiving position where you can exercise that skill. When learning a new programming language, try to do a small project it in before you have to do a large project. When learning to manage a software project, try to manage a small one first.

A good mentor is no replacement for doing things yourself, but is a lot better than a book. What can you offer a potential mentor in exchange for their knowledge? At a minimum, you should offer to study hard so their time won't be wasted.

Try to get your boss to let you have formal training, but understand that it often not much better than the same amount of time spent simply playing with the new skill you want to learn. It is, however, easier to ask for training than playtime in our imperfect world, even though a lot of formal training is just sleeping through lectures waiting for the dinner party.

If you lead people, understand how they learn and assist them by assigning them projects that are the right size and that exercise skills they are interested in. Don't forget that the most important skills for a programmer are not the technical ones. Give your people a chance to play and practice courage, honesty, and communication.
Learn to Type

Learn to touch-type. This is an intermediate skill because writing code is so hard that the speed at which you can type is irrelevant and can't put much of a dent in the time it takes to write code, no matter how good you are. However, by the time you are an intermediate programmer you will probably spend a lot of time writing natural language to your colleagues and others. This is a fun test of your commitment; it takes dedicated time that is not much fun to learn something like that. Legend has it that when Michael Tiemann[2] was at MCC people would stand outside his door to listen to the hum generated by his keystrokes which were so rapid as to be indistinguishable.
How to Do Integration Testing

Integration testing is the testing of the integration of various components that have been unit tested. Integration is expensive and it comes out in the testing. You must include time for this in your estimates and your schedule.

Ideally you should organize a project so that there is not a phase at the end where integration must explicitly take place. It is far better to gradually integrate things as they are completed over the course of the project. If it is unavoidable estimate it carefully.
Communication Languages

There are some languages, that is, formally defined syntactic systems, that are not programming languages but communication languages---they are designed specifically to facillitate communication through standardization. In 2003 the most important of these are UML, XML, and SQL. You should have some familiarity with all of these so that you can communicate well and decide when to use them.

UML is a rich formal system for making drawings that describe designs. It's beauty lines in that is both visual and formal, capable of conveying a great deal of information if both the author and the audience know UML. You need to know about it because designs are sometimes communicated in it. There are very helpful tools for making UML drawings that look very professional. In a lot of cases UML is too formal, and I find myself using a simpler boxes and arrows style for design drawings. But I'm fairly sure UML is at least as good for you as studying Latin.

XML is a standard for defining new standards. It is not a solution to data interchange problems, though you sometimes see it presented as if it was. Rather, it is a welcome automation of the most boring part of data interchange, namely, structuring the representation into a linear sequence and parsing back into a structure. It provides some nice type- and correctness-checking, though again only a fraction of what you are likely to need in practicen.

SQL is a very powerful and rich data query and manipulation language that is not quite a programming language. It has many variations, typically quite product-dependent, which are less important than the standardized core. SQL is the lingua franca of relational databases. You may or may not work in any field that can benefit from an understanding of relational databases, but you should have a basic understanding of them and they syntax and meaning of SQL.
Heavy Tools

As our technological culture progresses, software technology moves from inconceivable, to research, to new products, to standardized products, to widely available and inexpensive products. These heavy tools can pull great loads, but can be intimidating and require a large investment in understanding. The intermediate programmer has to know how to manage them and when they should be used or considered.

To my mind right some of the best heavy tools are:

*

Relational Databases,
*

Full-text Search Engines,
*

Math libraries,
*

OpenGL,
*

XML parsers, and
*

Spreadsheets.

How to analyze data
--

Data analysis is a process in the early stages of software development, when you examine a business activity and find the requirements to convert it into a software application. This is a formal definition, which may lead you to believe that data analysis is an action that you should better leave to the systems analysts, while you, the programmer, should focus on coding what somebody else has designed. If we follow strictly the software engineering paradigm, it may be correct. Experienced programmers become designers and the sharpest designers become business analysts, thus being entitled to think about all the data requirements and give you a well defined task to carry out. This is not entirely accurate, because data is the core value of every programming activity. Whatever you do in your programs, you are either moving around or modifying data. The business analyst is analyzing the needs in a larger scale, and the software designer is further squeezing such scale so that, when the problem lands on your desk, it seems that all you need to do is to apply clever algorithms and start moving existing data.

Not so.

No matter at which stage you start looking at it, data is the main concern of a well designed application. If you look closely at how a business analyst gets the requirements out of the customer?s requests, you?ll realize that data plays a fundamental role. The analyst creates so called Data Flow Diagrams, where all data sources are identified and the flow of information is shaped. Having clearly defined which data should be part of the system, the designer will shape up the data sources, in terms of database relations, data exchange protocols, and file formats, so that the task is ready to be passed down to the programmer. However, the process is not over yet, because you ? the programmer ? even after this thorough process of data refinement, are required to analyze data to perform the task in the best possible way. The bottom line of your task is the core message of Niklaus Wirth, the father of several languages. ?Algorithms + Data Structures = Programs.? There is never an algorithm standing alone, doing something to itself. Every algorithm is supposed to do something to at least one piece of data.

Therefore, since algorithms don't spin their wheels in a vacuum, you need to analyze both the data that somebody else has identified for you and the data that is necessary to write down your code. A trivial example will make the matter clearer. You are implementing a search routine for a library. According to your specifications, the user can select books by a combination of genre, author, title, publisher, printing year, and number of pages. The ultimate goal of your routine is to produce a legal SQL statement to search the back-end database. Based on these requirements, you have several choices: check each control in turn, using a "switch" statement, or several "if" ones; make an array of data controls, checking each element to see if it is set; create (or use) an abstract control object from which inherit all your specific controls, and connect them to an event-driven engine. If your requirements include also tuning up the query performance, by making sure that the items are checked in a specific order, you may consider using a tree of components to build your SQL statement. As you can see, the choice of the algorithm depends on the data you decide to use, or to create. Such decisions can make all the difference between an efficient algorithm and a disastrous one. However, efficiency is not the only concern. You may use a dozen named variables in your code and make it as efficient as it can ever be. But such a piece of code might not be easily maintainable. Perhaps choosing an appropriate container for your variables could keep the same speed and in addition allow your colleagues to understand the code better when they look at it next year. Furthermore, choosing a well defined data structure may allow them to extend the functionality of your code without rewriting it. In the long run, your choices of data determines how long your code will survive after you are finished with it. Let me give you another example, just some more food for thought. Let's suppose that your task is to find all the words in a dictionary with more than three anagrams, where an anagram must be another word in the same dictionary. If you think of it as a computational task, you will end up with an endless effort, trying to work out all the combinations of each word and then comparing it to the other words in the list. However, if you analyze the data at hand, you'll realize that each word may be represented by a record containing the word itself and a sorted array of its letters as ID. Armed with such knowledge, finding anagrams means just sorting the list on the additional field and picking up the ones that share the same ID. The brute force algorithm may take several days to run, while the smart one is just a matter of a few seconds. Remember this example the next time you are facing an intractable problem.
Team Skills
How to Manage Development Time

To manage development time, maintain a concise and up-to-date project plan. A project plan is an estimate, a schedule, a set of milestones for marking progress, and an assignment of your team or your own time to each task on the estimate. It should also include other things you have to remember to do, such as meeting with the quality assurance people, preparing documentation, or ordering equipment. If you are on a team, the project plan should be a consensual agreement, both at the start and as you go.

The project plan exists to help make decisions, not to show how organized you are. If the project plan is either too long or not up-to-date, it will be useless for making decisions. In reality, these decisions are about individual persons. The plan and your judgment let you decide if you should shift tasks from one person to another. The milestones mark your progress. If you use a fancy project planning tool, do not be seduced into creating a Big Design Up Front (BDUF) for the project, but use it maintain concision and up-to-dateness.

If you miss a milestone, you should take immediate action such as informing your boss that the scheduled completion of that project has slipped by that amount. The estimate and schedule could never have been perfect to begin with; this creates the illusion that you might be able to make up the days you missed in the latter part of the project. You might. But it is just as likely that you have underestimated that part as that you have overestimated it. Therefore the scheduled completion of the project has already slipped, whether you like it or not.

Make sure you plan includes time for: internal team meetings, demos, documentation, scheduled periodic activities, integration testing, dealing with outsiders, sickness, vacations, maintenance of existing products, and maintenance of the development environment. The project plan can serve as a way to give outsiders or your boss a view into what you or your team is doing. For this reason it should be short and up-to-date.
How to Manage Third-Party Software Risks

A project often depends on software produced by organizations that it does not control. There are great risks associated with third party software that must be recognized by everyone involved.

Never, ever, rest any hopes on vapor. Vapor is any alleged software that has been promised but is not yet available. This is the surest way to go out of business. It is unwise to be merely skeptical of a software company's promise to release a certain product with a certain feature at a certain date; it is far wiser to ignore it completely and forget you ever heard it. Never let it be written down in any documents used by your company.

If third-party software is not vapor, it is still risky, but at least it is a risk that can be tackled. If you are considering using third-party software, you should devote energy early on to evaluating it. People might not like to hear that it will take two weeks or two months to evaluate each of three products for suitability, but it has to be done as early as possible. The cost of integrating cannot be accurately estimated without a proper evaluation.

Understanding the suitability of existing third party software for a particular purpose is very tribal knowledge. It is very subjective and generally resides in experts. You can save a lot of time if you can find those experts. Often times a project will depend on a third-party software system so completely that if the integration fails the project will fail. Express risks like that clearly in writing in the schedule. Try to have a contingency plan, such as another system that can be used or the ability to write the functionality yourself if the risk can't be removed early. Never let a schedule depend on vapor.
How to Manage Consultants

Use consultants, but don't rely on them. They are wonderful people and deserve a great deal of respect. Since they get to see a lot of different projects, they often know more about specific technologies and even programming techniques than you will. The best way to use them is as educators in-house that can teach by example.

However, they usually cannot become part of the team in the same sense that regular employees are, if only because you may not have enough time to learn their strengths and weaknesses. Their financial commitment is much lower. They can move more easily. They may have less to gain if the company does well. Some will be good, some will be average, and some will be bad, but hopefully your selection of consultants will not be as careful as your selection of employees, so you will get more bad ones.

If consultants are going to write code, you must review it carefully as you go along. You cannot get to the end of the a project with the risk of a large block of code that has not been reviewed. This is true of all team members, really, but you will usually have more knowledge of the team members closer to you.
How to Communicate the Right Amount

Carefully consider the cost of a meeting; it costs its duration multiplied by the number of participants. Meetings are sometimes necessary, but smaller is usually better. The quality of communication in small meetings is better, and less time overall is wasted. If any one person is bored at a meeting take this as a sing, that the meeting should be smaller.

Everything possible should be done to encourage informal communication. More useful work is done during lunches with colleagues than during any other time. It is a shame that more companies do not recognize nor support this fact.
How to Disagree Honestly and Get Away with It

Disagreement is a great opportunity to make a good decision, but it should be handled delicately. Hopefully you feel that you have expressed your thoughts adequately and been heard before the decision is made. In that case there is nothing more to say, and you should decide whether you will stand behind the decision even though you disagree with it. If you can support this decision even though you disagree, say so. This shows how valuable you are because you are independent and are not a yes-man, but respectful of the decision and a team player.

Sometimes a decision that you disagree with will be made when the decision makers did not have the full benefit of you opinion. You should then evaluate whether to raise the issue on the basis of the benefit to the company or tribe. If it is a small mistake in your opinion, it may not be worth reconsidering. If it is a large mistake in you opinion, then of course you must present an argument.

Usually, this is not a problem. In some stressful circumstances and with some personality types this can lead to things being taken personally. For instance, some very good programmers lack the confidence needed to challenge a decision even when they have good reason to believe it is wrong. In the worst of circumstances the decision maker is insecure and takes it as a personal challenge to their authority. It is best to remember that in such circumstances people react with the reptilian part of their brains. You should present your argument in private, and try to show how new knowledge changes the basis on which the decision was made.

Whether the decision is reversed or not, you must remember that you will never be able to say ‘I told you so!’ since the alternate decision was fully explored.
Judgement
How to Tradeoff Quality Against Development Time

Software development is always a compromise between what the project does and getting the project done. But you may be asked to tradeoff quality to speed the deployment of a project in a way that offends your engineering sensibilities or business sensibilities. For example, you may be asked to do something that is a poor software engineering practice and that will lead to a lot of maintenance problems.

If this happens your first responsibility is to inform your team and to clearly explain the cost of the decrease in quality. After all, your understanding of it should be much better than your boss's understanding. Make it clear what is being lost and what is being gained, and at what cost the lost ground will be regained in the next cycle. In this, the visibility provided by a good project plan should be helpful. If the quality tradeoff affects the quality assurance effort, point that out (both to your boss and quality assurance people). If the quality tradeoff will lead to more bugs being reported after the quality assurance period, point that out.

If she still insists you should try to isolate the shoddiness into particular components that you can plan to rewrite or improve in the next cycle. Explain this to your team so that they can plan for it.

NinjaProgrammer at Slashdot sent in this gem:

Remember that a good design will be resillient against poor code implementations. If good interfaces and abstractions exist throughout the code, then the eventual rewrites will be far more painless. If it is hard to write clear code that is hard to fix, consider what is wrong with the core design that is causing this.

How to Manage Software System Dependence

Modern software systems tend to depend on a large number of components that may not be directly under your control. This increases productivity through synergy and reuse. However, each component brings with it some problems:

*

How will you fix bugs in the component?
*

Does the component restrict you to particular hardware or software systems?
*

What will you do if the component fails completely?

It is always best to encapsulate the component in some way so that it is isolated and so that it can be swapped out. If the component proves to be completely unworkable, you may be able to get a different one, but you may have to write your own. Encapsulation is not portability, but it makes porting easier, which is almost as good.

Having the source code for a component decreases the risk by a factor of four. With source code, you can evaluate it easier, debug it easier, find workarounds easier, and make fixes easier. If you make fixes, you should give them to the owner of the component and get the fixes incorporated into an official release; otherwise you will uncomfortably have to maintain an unofficial version .
How to Decide if Software is Too Immature

Using software other people wrote is one of the most effective ways to quickly build a solid system. It should not be discouraged, but the risks associated with it must be examined. One of the biggest risks is the period of bugginess and near inoperability that is often associated with software before it matures, through usage, into a usable product. Before you consider integrating with a software system, whether created in house or by a third party, it is very important to consider if it is really mature enough to be used. Here are ten questions you should ask yourself about it:

1.

Is it vapor? (Promises are very immature).
2.

Is there an accessible body of lore about the software?
3.

Are you the first user?
4.

Is there a strong incentive for continuation?
5.

Has it had a maintenance effort?
6.

Will it survive defection of the current maintainers?
7.

Is there a seasoned alternative at least half as good?
8.

Is it known to your tribe or company?
9.

Is it desirable to your tribe or company?
10.

Can you hire people to work on it even if it is bad?

A little consideration of these criteria demonstrates the great value of well-established free software and open-source software in reducing risk to the entrepreneur.
How to Make a Buy vs. Build Decision

An entrepreneurial company or project that is trying to accomplish something with software has to constantly make so-called buy vs. build decisions. This turn of phrase is unfortunate in two ways: it seems to ignore open-source and free software which is not necessarily bought. Even more importantly, it should perhaps be called an obtain and integrate vs. build here and integrate decision because the cost of integration must be considered. This requires a great combination of business, management, and engineering savvy.

*

How well do your needs match those for which it was designed?
*

What portion of what you buy will you need?
*

What is the cost of evaluating the integration?
*

What is the cost of integration?
*

What is the cost of evaluating the integration?
*

Will buying increase or decrease long term maintenance costs?
*

Will building it put you in a business position you don't want to be in?

You should think twice before building something that is big enough to serve as the basis for an entire other business. Such ideas are often proposed by bright and optimistic people that will have a lot to contribute to your team. If their idea is compelling, you may wish to change your business plan; but do not invest in a solution bigger than your own business without conscious thought.

After considering these questions, you should perhaps prepare two draft project plans, one for building and one for buying. This will force you to consider the integration costs. You should also consider the long term maintenance costs of both solutions. To estimate the integration costs, you will have to do a thorough evaluation of the software before you buy it. If you can't evaluate it, you will assume an unreasonable risk in buying it and you should decide against buying that particular product. If there are several buy decisions under consideration, some energy will have to be spent evaluating each.
How to Grow Professionally

Assume responsibility in excess of your authority. Play the role that you desire. Express appreciation for people's contribution to the success of the larger organization, as well as things as that help you personally.

If you want to become a team leader, instigate the formation of consensus. If you want to become a manager, take responsibility for the schedule. You can usually do this comfortably while working with a leader or a manager, since this frees them up to take greater responsibility. If that is too much to try, do it a little at a time.

Evaluate yourself. If you want to become a better programmer, ask someone you admire how you can become like them. You can also ask your boss, who will know less but have a greater impact on your career.

Plan ways to learn new skills, both the trivial technical kind, like learning a new software system, and the hard social kind, like writing well, by integrating them into your work.
How to Evaluate Interviewees

Evaluating potential employees is not given the energy it deserves. A bad hire, like a bad marriage, is terrible. A significant portion of everyone's energy should be devoted to recruitment, though this is rarely done.

There are different interviewing styles. Some are torturous, designed to put the candidate under a great deal of stress. This serves a very valuable purpose of possibly revealing character flaws and weaknesses under stress. Candidates are no more honest with interviewers than they are with themselves, and the human capacity for self-deception is astonishing.

You should, at a minimum, give the candidate the equivalent of an oral examination on the technical skills for two hours. With practice, you will be able to quickly cover what they know and quickly retract from what they don't know to mark out the boundary. Interviewees will respect this. I have several times heard interviewees say that the quality of the examination was one of their motivations for choosing a company. Good people want to be hired for their skills, not where they worked last or what school they went to or some other inessential characteristic.

In doing this, you should also evaluate their ability to learn, which is far more important than what they know. You should also watch for the whiff of brimstone that is given off by difficult people. You may be able to recognize it by comparing notes after the interview, but in the heat of the interview it is hard to recognize. How well people communicate and work with people is more important than being up on the latest programming language.

A reader has had good luck using a ‘take-home’ test for interviewees. This has the advantage that can uncover the interviewee that can present themselves well but can't really code---and there are many such people. I personally have not tried this technique, but it sounds sensible.

Finally, interviewing is also a process of selling. You should be selling your company or project to the candidate. However, you are talking to a programmer, so don't try to color the truth. Start off with the bad stuff, then finish strong with the good stuff.
How to Know When to Apply Fancy Computer Science

There is a body of knowledge about algorithms, data structures, mathematics, and other gee-whiz stuff that most programmers know about but rarely use. In practice, this wonderful stuff is too complicated and generally unnecessary. There is no point in improving an algorithm when most of your time is spent making inefficient database calls, for instance. An unfortunate amount of programming consists of getting systems to talk to each other and using very simple data structures to build a nice user interface.

When is high technology the appropriate technology? When should you crack a book to get something other than a run-of-the-mill algorithm? It is sometimes useful to do this but it should be evaluated carefully.

The three most important considerations for the potential computer science technique are:

*

Is it well encapsulated so that the risk to other systems is low and the overall increase in complexity and maintenance cost is low?
*

Is the benefit startling (for example, a factor of two in a mature system or a factor of ten in a new system)?
*

Will you be able to test and evaluate it effectively?

If a well-isolated algorithm that uses a slightly fancy algorithm can decrease hardware cost or increase performance by a factor of two across an entire system, then it would be criminal not to consider it. One of the keys to arguing for such an approach is to show that the risk is really quite low, since the proposed technology has probably been well studied, the only issue is the risk of integration. Here a programmer's experience and judgment can truly synergize with the fancy technology to make integration easy.
How to Talk to Non-Engineers

Engineers and programmers in particular are generally recognized by popular culture as being different from other people. This implies that other people are different from us. This is worth bearing in mind when communicating with non-engineers; you should always understand the audience.

Non-engineers are smart, but not as grounded in creating technical things as we are. We make things. They sell things and handle things and count things and manage things, but they are not experts on making things. They are not as good at working together on teams as engineers are (there are no doubt exceptions.)[3] Their social skills are generally as good as or better than engineers in non-team environments, but their work does not always demand that they practice the kind of intimate, precise communication and careful subdivisions of tasks that we do.

Non-engineers may be too eager to please and they may be intimidated by you. Just like us, they may say ‘yes’ without really meaning it to please you or because they are a little scared of you, and then not stand behind their words.

Non-programmers can understand technical things but they do not have the thing that is so hard even for us---technical judgment. They do understand how technology works, but they cannot understand why a certain approach would take three months and another one three days. (After all, programmers are anecdotally horrible at this kind of estimation as well.) This represents a great opportunity to synergize with them.

When talking to your team you will, without thinking, use a sort of shorthand, an abbreviated language that is effective because you will have much shared experience about technology in general and your product in particular. It takes some effort not to use this shorthand with those that don't have that shared experience, especially when members of your own team are present. This vocabulary create a wall between you and those that do not share it, and, even worse, wastes their time.

With your team, the basic assumptions and goals do not need to be restated often, and most conversation focuses on the details. With outsiders, it must be the other way around. They may not understand things you take for granted. Since you take them for granted and don't repeat them, you can leave a conversation with an outsider thinking that you understand each other when really there is a large misunderstanding. You should assume that you will miscommunicate and watch carefully to find this miscommunication. Try to get them to summarize or paraphrase what you are saying to make sure they understand. If you have the opportunity to meet with them often, spend a little bit of time asking if you you are communicating effectively, and how you can do it better. If there is a problem in communication, seek to alter your own practices before becoming frustrated with theirs.

I love working with non-engineers. It provides great opportunities to learn and to teach. You can often lead by example, in terms of the clarity of your communication. Engineers are trained to bring order out of chaos, to bring clarity out of confusion, and non-engineers like this about us. Because we have technical judgment and can usually understand business issues, we can often find a simple solution to a problem.

Often non-engineers propose solutions that they think will make it easier on us out of kindness and a desire to do the right thing, when in fact a much better overall solution exists which can only be seen by synergizing the outsiders view with your technical judgment. I personally like Extreme Programming because it addresses this inefficiency; by marrying the estimation quickly to the idea, it makes it easier to find the idea that is the best combination of cost and benefit.

[1] “to hit”
Chapter 4. Advanced

Table of Contents

Technological Judgment

How to Tell the Hard From the Impossible
How to Utilize Embedded Languages
Choosing Languages

Compromising Wisely

How to Fight Schedule Pressure
How to Understand the User
How to Get a Promotion

Serving Your Team

How to Develop Talent
How to Choose What to Work On
How to Get the Most From Your Teammates
How to Divide Problems Up
How to Handle Boring Tasks
How to Gather Support for a Project
How to Grow a System
How to Communicate Well
How to Tell People Things They Don't Want to Hear
How to Deal with Managerial Myths
How to Deal with Organizational Chaos

Technological Judgment
How to Tell the Hard From the Impossible

It is our job to do the hard and discern the impossible. From the point of view of most working programmers, something is impossible if either it cannot be grown from a simple system or it cannot be estimated. By this definition what is called research is impossible. A large volume of mere work is hard, but not necessarily impossible.

The distinction is not facetious because you may very well be asked to do what is practically impossible, either from a scientific point of view or a software engineering point of view. It then becomes your job to help the entrepreneur find a reasonable solution which is merely hard and gets most of what they wanted. A solution is merely hard when it can be confidently scheduled and the risks are understood.

It is impossible to satisfy a vague requirement, such as ‘Build a system that will compute the most attractive hair style and color for any person.’ If the requirement can be made more crisp, it will often become merely hard, such as ‘Build a system to compute an attractive hair style and color for a person, allow them to preview it and make changes, and have the customer satisfaction based on the original styling be so great that we make a lot of money.’ If there is not crisp definition of success, you will not succeed.
How to Utilize Embedded Languages

Embedding a programming language into a system has an almost erotic fascination to a programmer. It is one of the most creative acts that can be performed. It makes the system tremendously powerful. It allows you to exercise her most creative and Promethean skills. It makes the system into your friend.

The best text editors in the world all have embedded languages. This can be used to the extent that the intended audience can master the language. Of course, use of the language can be made optional, as it is in text editors, so that initiates can use it and no one else has to.

I and many other programmers have fallen into the trap of creating special purpose embedded languages. I fell into it twice. There already exist many languages designed specifically to be embedded languages. You should think twice before creating a new one.

The real question to ask oneself before embedding a language is: Does this work with or against the culture of my audience? If you intended audience is exclusively non-programmers, how will it help? If your intended audience is exclusively programmers, would they prefer an applications programmers interface (API)? And what language will it be? Programmers don't want to learn a new language that is narrowly used; but if it meshes with their culture they will not have to spend much time learning it. It is a joy to create a new language. But we should not let that blind us to the needs of the user. Unless you have some truly original needs and ideas, why not use an existing language so that you can leverage the familiarity users already have with it?
Choosing Languages

The solitary programmer that loves his work (a hacker) can choose the best language for the task. Most working programmers have very little control of the language they will use. Generally, this issue is dictated by pointy-haired bosses who are making a political decision, rather than a technological decision, and lack the courage to promote an unconventional tool even when they know, often with firsthand knowledge, that the less accepted tool is best. In other cases the very real benefit of unity among the team, and to some extent with a larger community, precludes choice on the part of the individual. Often managers are driven by the need to be able to hire programmers with experience in a given language. No doubt they are serving what they perceive to be the best interests of the project or company, and must be respected for that. However, I personally believe this the most wasteful and erroneous common practice you are likely to encounter.

But of course, things are never one-dimensional. Even if a core language is mandated and beyond your control, it is often the case that tools and other programs can and should be written in a different language. If a language is to be embedded (and you should always consider it!) the choice of language will depend a lot on the culture of the users. One should take advantage of this to serve your company or project by using the best language for the job, and in so doing make work more interesting.

Programming languages should really be called notations in that learning one is not at all as difficult as learning a natural language. To beginners and to some outsiders ``learning a new language'' seems a daunting task; but after you have three under your belt it's really just a question of becoming familiar with the available libraries. One tends to think of a large system that has components in three or four languages as a messy hodgepodge; but I argue that such a system is in many cases stronger than a one-language system in several ways:

*

There is necessarily loose coupling between the components that are written in different notations (though maybe not clean interfaces),
*

You can evolve to a new language/platform easily by rewriting each component individually,
*

Its possible that some of the modules are actually up-to-date.

Some of these effects may only be psychological; but psychology matters. In the end the costs of language tyranny outweigh any advantage that it provides.
Compromising Wisely
How to Fight Schedule Pressure

Time-to-market pressure is the pressure to deliver a good product quickly. It is good because it reflects a financial reality, and is healthy up to a point. Schedule pressure is the pressure to deliver something faster than it can be delivered and it is wasteful, unhealthy, and all too common.

Schedule pressure exists for several reasons. The people who task programmers do not fully appreciate what a strong work ethic we have and how much fun it is to be a programmer. Perhaps because they project their own behavior onto us, they believe that asking for it sooner will make us work harder to get it there sooner. This is probably actually true, but the effect is very small, and the damage is very great. Additionally, they have no visibility into what it really takes to produce software. Not being able to see it, and not be able to create it themselves, the only thing they can do is see time-to-market pressure and fuss at programmers about it.

The key to fighting schedule pressure is simply to turn it into time-to-market pressure. The way to do this to give visibility into the relationship between the available labor and the product. Producing an honest, detailed, and most of all, understandable estimate of all the labor involved is the best way to do this. It has the added advantage of allowing good management decisions to be made about possible functionality tradeoffs.

The key insight that the estimate must make plain is that labor is an almost incompressible fluid. You can't pack more into a span of time anymore than you can pack more water into a container over and above that container's volume. In a sense, a programmer should never say ‘no’, but rather to say ‘What will you give up to get that thing you want?’ The effect of producing clear estimates will be to increase the respect for programmers. This is how other professionals behave. Programmers' hard work will be visible. Setting an unrealistic schedule will also be painfully obvious to everyone. Programmers cannot be hoodwinked. It is disrespectful and demoralizing to ask them to do something unrealistic. Extreme Programming amplifies this and builds a process around it; I hope that every reader will be lucky enough to use it.
How to Understand the User

It is your duty to understand the user, and to help your boss understand the user. Because the user is not as intimately involved in the creation of your product as you are, they behave a little differently:

*

The user generally makes short pronouncements.
*

The user has their own job; they will mainly think of small improvements in your product, not big improvements.
*

The user can't have a vision that represents the complete body of your product users.

It is your duty to give them what they really want, not what they say they want. It is however, better to propose it to them and get them to agree that your proposal is what they really want before you begin, but they may not have the vision to do this. Your confidence in your own ideas about this should vary. You must guard against both arrogance and false modesty in terms of knowing what the customer really wants. Programmers are trained to design and create. Market researchers are trained to figure out what people want. These two kinds of people, or two modes of thought in the same person, working harmoniously together give the best chance of formulating the correct vision.

The more time you spend with users the better you will be able to understand what will really be successful. You should try to test your ideas against them as much as you can. You should eat and drink with them if you can.

Guy Kawasaki[Rules] has emphasized the importance of watching what your users do in addition to listening to them.

I believe contractors and consultants often have tremendous problems getting their clients to clarify in their own minds what they really want. If you intend to be a consultant, I suggest you choose your clients based on their clear-headedness as well as their pocketbooks.
How to Get a Promotion

To be promoted to a role, act out that role first.

To get promoted to a title, find out what is expected of that title and do that.

To get a pay raise, negotiate armed with information.

If you feel like you are past due for a promotion, talk to your boss about it. Ask them explicitly what you need to do to get promoted, and try to do it. This sounds trite, but often times your perception of what you need to do will differ considerably from your boss's. Also this will pin your boss down in some ways.

Most programmers probably have an exaggerated sense of their relative abilities in some ways---after all, we can't all be in the top 10%! However, I have seem some people who were seriously unappreciated. One cannot expect everyone's evaluation to perfectly match reality at all times, but I think people are generally moderately fair, with one caveat: you cannot be appreciated without visibility into your work. Sometimes, do to happenstance or personal habits, someone will not be noticed much. Working from home a lot or being geographically separated from your team and boss makes this especially difficult.
Serving Your Team
How to Develop Talent

Nietschze exaggerated when he said[Stronger]:

What does not destroy me, makes me stronger.

Your greatest responsibility is to your team. You should know each of them well. You should stretch your team, but not overburden them. You should usually talk to them about the way they are being stretched. If they buy in to it, they will be well motivated. On each project, or every other project, try to stretch them in both a way that they suggest and a way that you think will be good for them. Stretch them not by giving them more work, but by giving them a new skill or better yet a new role to play on the team.

You should allow people (including yourself) to fail occasionally and should plan for some failure in your schedule. If there is never any failure, there can be no sense of adventure. If there are not occasional failures, you are not trying hard enough. When someone fails, you should be as gentle as you can with them while not treating them as though they had succeeded.

Try to get each team member to buy in and be well motivated. Ask each of them explicitly what they need to be well-motivated if they are not. You may have to leave them dissatisfied, but you should know what everybody desires.

You can't give up on someone who is intentionally not carrying their share of the load because of low morale or dissatisfaction and just let them be slack. You must try to get them well-motivated and productive. As long as you have the patience, keep this up. When your patience is exhausted, fire them. You cannot allow someone who is intentionally working below their level to remain on the team, since it is not fair to the team.

Make it clear to the strong members of your team that you think they are strong by saying so in public. Praise should be public and criticism private.

The strong members of the team will naturally have more difficult tasks than the weak members of the team. This is perfectly natural and nobody will be bothered by it as long as everyone works hard.

It is an odd fact that is not reflected in salaries that a good programmer is more productive than 10 bad programmers. This creates a strange situation. It will often be true that you could move faster if your weak programmers would just get out of the way. If you did this you would in fact make more progress in the short term. However, your tribe would lose some important benefits, namely the training of the weaker members, the spreading of tribal knowledge, and the ability to recover from the loss of the strong members. The strong must be gentle in this regard and consider the issue from all angles.

You can often give the stronger team members challenging, but carefully delineated, tasks.
How to Choose What to Work On

You balance your personal needs against the needs of the team in choosing what aspect of a project to work on. You should do what you are best at, but try to find a way to stretch yourself not by taking on more work but by exercising a new skill. Leadership and communication skills are more important than technical skills. If you are very strong, take on the hardest or riskiest task, and do it as early as possible in the project to decrease risk.
How to Get the Most From Your Teammates

To get the most from your teammates, develop a good team spirit and try to keep every individual both personally challenged and personally engaged.

To develop team spirit, corny stuff like logoized clothing and parties are good, but not as good as personal respect. If everyone respects everyone else, nobody will want to let anybody down. Team spirit is created when people make sacrifices for the team and think in terms of the good of the team before their own personal good. As a leader, you can't ask for more than you give yourself in this respect.

One of the keys to team leadership is to facilitate consensus so that everyone has buy in. This occasionally means allowing your teammates to be wrong. That is, if it does not harm the project too much, you must let some of your team do things their own way, based on consensus, even if you believe with great confidence it is the wrong thing to do. When this happens, don't agree, simply disagree openly and accept the consensus. Don't sound hurt, or like you're being forced into it, simply state that you disagree but think the consensus of the team is more important. This will often cause them to backtrack. Don't insist that they go through with their initial plan if they do backtrack.

If there is an individual who will not consent after you have discussed the issues from all appropriate sides, simply assert that you have to make a decision and that is what your decision is. If there is a way to judge if your decision will be wrong or if it is later shown to be wrong, switch as quickly as you can and recognize the persons who were right.

Ask your team, both as a group and individually, what they think would create team spirit and make for an effective team.

Praise frequently rather than lavishly. Especially praise those who disagree with you when they are praiseworthy. Praise in public and criticize in private; with one exception: sometimes growth or the correction of a fault can't be praised without drawing embarrassing attention to the original fault, so that growth should be praised in private.
How to Divide Problems Up

It's fun to take a software project and divide it up into tasks that will be performed by individuals. This should be done early. Sometimes managers like to think that an estimate can be made without consideration of the individuals that will perform the work. This is impossible since the productivity of individuals varies so widely. Who has particular knowledge about a component also constantly changes and can have an order of magnitude effect on performance.

Just as a composer considers the timbre of the instrument that will play a part or the coach of an athletic team considers the strengths of each player, the experienced team leader will not usually be able to separate the division of the project into tasks from the team members to which they will be assigned. This is part of the reason that a high-performing team should not be broken up.

There is a certain danger in this given that people will become bored as they build upon their strengths and never improve their weaknesses or learn new skills. However, specialization is a very useful productivity tool when not overused.
How to Handle Boring Tasks

Sometimes it is not possible to avoid boring tasks that are critical to the success of the company or the project. These tasks can really hurt the morale of those that have to do them. The best technique for dealing with this is to invoke or promote Larry Wall's programmer's virtue of Laziness. Try to find some way to get the computer to do the task for you or to help your teammates do this. Working for a week on a program to do a task that will take a week to do by hand has the great advantage of being more educational and sometimes more repeatable.

If all else fails, apologize to those who have to do the boring task, but under no circumstances allow them to do it alone. At a minimum assign a team of two to do the work and promote healthy teamwork to get the task done.
How to Gather Support for a Project

To gather support for a project, create and communicate a vision that demonstrates real value to the organization as a whole. Attempt to let others share in your vision creation. This gives them a reason to support you and gives you the benefit of their ideas. Individually recruit key supporters for your project. Wherever possible, show, don't tell. If possible, construct a prototype or a mockup to demonstrate your ideas. A prototype is always powerful but in software it is far superior to any written description.
How to Grow a System

The seed of a tree contains the idea of the adult but does not fully realize the form and potency of the adult. The embryo grows. It becomes larger. It looks more like the adult and has more of the uses. Eventually it bears fruit. Later, it dies and its body feeds other organisms.

We have the luxury of treating software like that. A bridge is not like that; there is never a baby bridge, but merely an unfinished bridge. Bridges are a lot simpler than software.

It is good to think of software as growing, because it allows us to make useful progress before we have a perfect mental image. We can get feedback from users and use that to correct the growth. Pruning off weak limbs is healthful.

The programmer must design a finished system that can be delivered and used. But the advanced programmer must do more. You must design a growth path that ends in the finished system. It is your job to take a germ of an idea and build a path that takes it as smoothly as possible into a useful artifact.

To do this, you must visualize the end result and communicate it in a way that the engineering team can get excited about. But you must also communicate to them a path that goes from wherever they are now to where they want to be with no large leaps. The tree must stay alive the whole time; it cannot be dead at one point and resurrected later.

This approach is captured in spiral development. Milestones that are never too far apart are used to mark progress along the path. In the ultra-competitive environment of business, it is best if the milestones can be released and make money as early as possible, even if they are far away from a well-designed endpoint. One of the programmer's jobs is to balance the immediate payoff against future payoff by wisely choosing a growth path expressed in milestones.

The advanced programmer has the triple responsibility of growing software, teams, and persons.

A reader, Rob Hafernik, sent in this comment on this section that I can do no better than to quote in full:

I think you under-emphasize the importance here. It's not just systems, but algorithms, user interfaces, data models, and so on. It's absolutely vital as you work on a large system to have measurable progress toward intermediate goals. Nothing is as bad as the special horror of getting down to the end and discovering that the whole thing just isn't going to work (look at the recent debacle of the Voter News System). I would even go further and state it as a law of nature: no large, complex system can be implemented from scratch, it can only be evolved from a simple system to a complex system in a series of intentional steps.

To which one can only reply Fiat lux!
How to Communicate Well

To communicate well, you have to recognize how hard it is. It is a skill unto itself. It is made harder by the fact that the persons with whom you have to communicate are flawed. They do not work hard at understanding you. They speak poorly and write poorly. They are often overworked or bored, and, at a minimum, somewhat focused on their own work rather than the larger issues you may be addressing. One of the advantages of taking classes and practicing writing, public speaking, and listening is that if you become good at it you can more readily see where problems lie and how to correct them.

The programmer is a social animal whose survival depends on communication with her team. The advanced programmer is a social animal whose satisfaction depends on communication with people outside her team.

The programmer brings order out of chaos. One interesting way to do this is to initiate a proposal of some kind outside the team. This can be done in a strawman or white-paper format or just verbally. This leadership has the tremendous advantage of setting the terms of the debate. It also exposes you to criticism, and worse, rejection and neglect. The advanced programmer must be prepared to accept this, because she has a unique power and therefore a unique responsibility. Entrepreneurs who are not programmers need programmers to provide leadership in some ways. Programmers are the part of the bridge between ideas and reality that rests on reality.

I haven't mastered communicating well, but what I'm currently trying is what I think of a four-pronged approach: After I have my ideas in order and am fully prepared, I try to speak verbally, hand people a white-paper (on real paper, as well as electronically) show them a demo, and then patiently repeat this process. I think a lot of times we are not patient enough in this kind of difficult communication. You should not be disheartened if your ideas are not immediately accepted. If you have invested energy in there preparation, no one will think poorly of you for it.
How to Tell People Things They Don't Want to Hear

You will often have to tell people things that will make them uncomfortable. Remember that you are doing this for a reason. Even if nothing can be done about the problem, you are telling them as early as possible so they will be well-informed.

The best way to tell someone about a problem is to offer a solution at the same time. The second best way is to appeal to them for help with the problem. If there is a danger that you won't be believed, you should gather some support for your assertion.

One of the most unpleasant and common things you will have to say is, ‘The schedule will have to slip.’ The conscientious programmer hates to say this, but must say it as early as possible. There is nothing worse than postponing action when a milestone slips, even if the only action is to inform everyone. In doing this, it is better to do it as a team, at least in spirit, if not physically. You will want your team's input on both where you stand and what can be done about it, and the team will have to face the consequences with you.
How to Deal with Managerial Myths

The word myth sometimes means fiction. But it has a deeper connotation. It also means a story of religious significance that explains the universe and mankind's relationship to it. Managers tend to forget what they learned as programmers and believe in certain myths. It would be as rude and unsuccessful to try to convince them these myths are false as to try to disillusion a devoutly religious person of their beliefs. For that reason, you should recognize these beliefs as myths:

*

More documentation is always better. (They want it, but they don't want you to spend any time on it.)
*

Programmers can be equated. (Programmers vary by an order of magnitude.)
*

Resources can be added to a late project to speed it. (The cost of communication with the new persons is almost always more taxing than helpful.)
*

It is possible to estimate software development reliably. (It is not even theoretically possible.)
*

Programmers' productivity can be measured in terms of some simple metric, like lines of code. (If succinctness is power, lines of code are bad, not good.)

If you have an opportunity, you can try to explain these things, but don't feel bad if you have no success and don't damage your reputation by confronting these myths belligerently. Each of these myths reinforces the manager's idea that they have some actual control over what is going on. The truth is that managers facilitate if they are good, and impede if they are bad.
How to Deal with Organizational Chaos

There are often brief periods of great organizational chaos, such as layoffs, buyouts, ipos, firings, new hirings, and so on. These are unsettling to everyone, but perhaps a little less unsettling to the programmer whose personal self-esteem is founded in capacity rather than in position. Organizational chaos is a great opportunity for programmers to exercise their magic power. I've saved this for last because it is a deep tribal secret. If you are not a programmer, please stop reading now.

Engineers have the power to create and sustain.

Non-engineers can order people around but, in a typical software company, can create and sustain nothing without engineers, just as engineers typically cannot sell a product or manage a business effectively. This power is proof against almost all of the problems associated with temporary organizational mayhem. When you have it you should ignore the chaos completely and carry on as if nothing is happening. You may, of course, get fired, but if that happens you can probably get a new job because of the magic power. More commonly, some stressed-out person who does not have the magic power will come into your cube and tell you to do something stupid. If you are really sure that it is stupid, it is best to smile and nod until they go away and then carry on doing what you know is best for the company.

If you are a leader, tell your people to do the same thing and tell them to ignore what anybody else tells them. This course of action is the best for you personally, and is the best for your company or project.
Glossary

This is a glossary of terms as used in this essay. These do not necessarily have a standardized meaning to other people. Eric S. Raymond has compiled a massive and informative glossary[HackerDict] that rather surprisingly can pleasurably be read cover-to-cover once you can appreciate a fraction of it.

unk-unk

Slang for unknown-unknown. Problems that cannot presently even be conceptualized that will steal time away from the project and wreck the schedule.
boss

The person or entity that gives you tasks. In some cases this may be the public at large.
printlining

The insertion of statements into a program on a strictly temporary basis that output information about the execution of the program for the purpose of debugging.
logging

The practice of writing a program so that it can produce a configurable output log describing its execution.
divide and conquer

A technique of top-down design and, importantly, of debugging that is the subdivision of a problem or a mystery into progressively smaller problems or mysteries.
vapor

Illusionary and often deceptive promises of software that is not yet for sale and, as often as not, will never materialize into anything solid.
boss

The person who sets your tasks. In some cases, the user is the boss.
tribe

The people with whom you share loyalty to a common goal.
low-hanging fruit

Big improvements that cost little.
Entrepreneur

The initiator of projects.
garbage

Objects that are no longer needed that hold memory.
busines

A group of people organized for making money.
company

A group of people organized for making money.
tribe

A group of people you share cultural affinity and loyalty with.
scroll blindness

The effect of being unable to find information you need because it is buried in too much other, less interesting information.
wall-clock

Actually time as measured by a clock on a wall, as opposed to CPU time.
bottleneck

The most important limitation in the performance of a system. A constriction that limits performance.
master

A unique piece of information from which all cached copies are derived that serves as the official definition of that data.
heap allocated

Memory can be said to be heap allocated whenever the mechanism for freeing it is complicated.
garbage

Allocated memory that no longer has any useful meaning.
garbage collector

A system for recycling garbage.
memory leak

The unwanted collection of references to objects that prevents garbage collection (or a bug in the garbage collector or memory management system!) that causes the program to gradually increase its memory demands over time.
Extreme Programming

A style of programming emphasizing communication with the customer and automated testing.
hitting the wall

To run out of a specific resource causing performance to degrade sharply rather than gradually.
speculative programming

Producing a feature before it is really known if that feature will be useful.
information hiding

A design principle that seeks to keep things independent and decoupled by using interfaces that expose as little information as possible.
object-oriented programming

An programming style emphasizing the the management of state inside objects.
communication languages

A language designed primarily for standardization rather than execution.
boxes and arrows

A loose, informal style of making diagrams consiting of boxes and arrows drawn between those boxes to show the relationships. This contrast with formal diagram methodologies, such as UML.
lingua franca

A language so popular as to be a de facto standard for its field, as French was for international diplomacy at one time.
buy vs. build

An adjective describing a choice between spending money for software or writing it your self.
mere work

Work that requires little creativity and entails little risk. Mere work can be estimated easily.
programming notation

A synonym for programming language that emphasizes the mathematical nature of programming language and their relative simplicity compared to natural languages.
strawman

A document meant to be the starting point of a technical discussion. A strawman may lead to a stickman, tinman, woodman, ironman, etc.
wite-paper

An informative document that is often meant to explain or sell a product or idea to an audience different than the programmers of that product or idea.

Table of Contents

Bibliography/Websiteography

Bibliography/Websiteography
Books

[Rules00] Guy Kawasaki, Michelle Moreno, and Gary Kawasaki. 2000. HarperBusiness. Rules for Revolutionaries: The Capitalist Manifesto for Creating and Marketing New Products and Services.

[RDev96] Steve McConnell. 1996. Microsoft Press. Redmond, Wash. Rapid Development: Taming Wild Software Schedules.

[CodeC93] Steve McConnell. 1993. Microsoft Press. Redmond, Wash. Code Complete.

[XP99] Kent Beck. 1999. 0201616416. Addison-Wesley. Extreme Programming Explained: Embrace Change.

[PlanXP00] Kent Beck and Martin Fowler. 2000. 0201710919. Addison-Wesley. Planning Extreme Programming.

[Prag99] Andrew Hunt, David Thomas, and Ward Cunningham. 1999. 020161622X. Addison-Wesley. The Pragmatic Programmer: From Journeyman to Master.

[Stronger] Friedrich Nietzsche. 1889. Twilight of the Idols, "Maxims and Arrows", section 8..
Web Sites

[PGSite] Paul Graham. 2002. Articles on his website: http://www.paulgraham.com/articles.html. All of them, but especially "Beating the Averages".

[Hacker] Eric S. Raymond. 2003. How to Become a Hacker. http://www.catb.org/~esr/faqs/hacker-howto.html.

[HackDict] Eric S. Raymond. 2003. The New Hacker Dictionary. http://catb.org/esr/jargon/jargon.html.

[ExpCS] Edsger W. Dijkstra. 1986. How Experimental is Computing Science?. http://www.cs.utexas.edu/users/EWD/ewd09xx/EWD988a.PDF.

[Knife] Edsger W. Dijkstra. 1984. On a Cultural Gap. http://www.cs.utexas.edu/users/EWD/ewd09xx/EWD913.PDF .
Appendix B. History (As Of February, 2003)
Request for Feedback or Extension

Please send me any comments you may have on this essay. I consider all suggestions, many of which have already improved this essay.

I have placed this essay under the GNU Free Documentation License. This license is not specifically designed for essays. Essays are usually intended to be coherent and convincing arguments that are writtien from a single point of view in a single voice. I hope this essay is a short and pleasant read.

I also hope that it is instructive. Although not a textbook, it is broken into many small sections to which new sections can be freely added. If so inclined, you are encouraged to expand upon this essay as you see fit, subject to the provisions of the License.

It may be arrogance to imagine that this document is worthy of extension; but hope springs eternal. I would be joyous if it were extended in the following ways:

*

The addition of a comprehensive reading list to each section,
*

The addition of more and better sections,
*

Translation into other languages, even if only on a subsection-by-subsection basis, and/or
*

Criticism or commentary in-lined into the text.
*

The ability to build into different formats, such as palm formats and better HTML.

If you inform me of your work, I will consider it and may include it in subsequent versions that I produce, subject to the provisions of the License. You may of course produce your own versions of this document without my knowledge, as explained in the License.

Thank you.

Robert L. Read
Original Version

The original version of this document was begun by Robert L. Read in the year 2000 and first published electronically at Samizdat Press(http://Samizdat.mines.edu) in 2002. It is dedicated to the programmers of Hire.com.

After this article was mentioned on Slashdot in 2003, about 75 people sent me email with suggestions and errata. I appreciate them all. There was a lot of duplication, but the following people either made major suggestions or were the first to find a bug that I fixed: Morgan McGuire, David Mason, Tom Moertel, Ninja Programmer (145252) at Slashdot, Ben Vierck, Rob Hafernik, Mark Howe, Pieter Pareit, Brian Grayson, Zed A. Shaw, Steve Benz, Maksim Ioffe, Andrew Wu, David Jeschke, and Tom Corcoran.

Finally I would like to thank Christina Vallery, whose editing and proofreading greatly improved the second draft, and Wayne Allen, who encouraged me to initiate this.
Original Author's Bio

Robert L. Read lives in Austin, Texas, with his wife and two children. He is currently a Principal Engineer at Hire.com, where he has worked for four years. Prior to that he founded 4R Technology, which made a scanner-based image analysis quality control tool for the paper industry.

Rob received a PhD from the University of Texas at Austin in 1995 in Computer Science related to database theory. In 1987 he received a BA in Computer Science from Rice University. He has been a paid programmer since the age of 16.
Appendix C. GNU Free Documentation License
Version 1.2, November 2002

Table of Contents

PREAMBLE
APPLICABILITY AND DEFINITIONS
VERBATIM COPYING
COPYING IN QUANTITY
MODIFICATIONS
COMBINING DOCUMENTS
COLLECTIONS OF DOCUMENTS
AGGREGATION WITH INDEPENDENT WORKS
TRANSLATION
TERMINATION
FUTURE REVISIONS OF THIS LICENSE
ADDENDUM: How to use this License for your documents

Copyright (C) 2000,2001,2002 Free Software Foundation, Inc. 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

PREAMBLE

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.

This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.

We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.
APPLICABILITY AND DEFINITIONS

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.

A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.

A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.

The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.

The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.

A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".

Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.

The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.

A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.

The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.
VERBATIM COPYING

You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.

You may also lend copies, under the same conditions stated above, and you may publicly display copies.
COPYING IN QUANTITY

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.

If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.

If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.

It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.
MODIFICATIONS

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:

1. Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.
2. List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.
3. State on the Title page the name of the publisher of the Modified Version, as the publisher.
4. Preserve all the copyright notices of the Document.
5. Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.
6. Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.
7. Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice.
8. Include an unaltered copy of this License.
9. Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.
10. Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.
11. For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.
12. Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.
13. Delete any section Entitled "Endorsements". Such a section may not be included in the Modified Version.
14. Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant Section.
15. Preserve any Warranty Disclaimers.

If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.

You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties--for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.

You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.

The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.
COMBINING DOCUMENTS

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.

The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.

In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".
COLLECTIONS OF DOCUMENTS

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.

You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.
AGGREGATION WITH INDEPENDENT WORKS

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation's users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.

If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.
TRANSLATION

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.

If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.
TERMINATION

You may not copy, modify, sublicense, or distribute the Document except as expressly provided for under this License. Any other attempt to copy, modify, sublicense or distribute the Document is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.
FUTURE REVISIONS OF THIS LICENSE

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.

Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation.
ADDENDUM: How to use this License for your documents

To use this License in a document you have written, include a copy of the License in the document and put the following copyright and license notices just after the title page:

Copyright (c) YEAR YOUR NAME. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".

If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the "with...Texts." line with this:

with the Invariant Sections being LIST THEIR TITLES, with the Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.

If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.

If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.
Colophon

This work is licensed under the GNU Free Documentation License. It was originally written in LaTeX, then ported to DocBook. It is available in transparent copy in the DocBook/XML format. A build system and special instructions on processing are part of the transparent copy package, though building from the source is system dependent. The source can be built into either HTML or PDF format, and extension to other formats should be possible and is encouraged.

The original version of this document was written by Robert L. Read without renumeration and dedicated to the programmers of Hire.com.

----------------------------------------------------------









A Brief History of Hackerdom
by Eric S. Raymond
$Date: 2000/05/05 18:57:21 $

I explore the origins of the hacker culture, including prehistory among the Real Programmers, the glory days of the MIT AI Lab, and how the early ARPANET nurtured the first network nation. Storm clouds over Jupiter. I describe the early rise and eventual stagnation of Unix, the new hope from Finland, and how `the last true hacker' became the next generation's patriarch. I sketch the way Linux and the mainstreaming of the Internet brought the hacker culture from the fringes of public consciousness to its current prominence.

1. Translations

2. Prologue: The Real Programmers

3. The Early Hackers

4. The Rise of Unix

5. The End of Elder Days

6. The Proprietary-Unix Era

7. The Early Free Unixes

8. The Great Web Explosion

9. Bibliography





2. Prologue: The Real Programmers

In the beginning, there were Real Programmers.

That's not what they called themselves. They didn't call themselves `hackers', either, or anything in particular; the sobriquet `Real Programmer' wasn't coined until after 1980, retrospectively by one of their own. But from 1945 onward, the technology of computing attracted many of the world's brightest and most creative minds. From Eckert & Mauchly's first ENIAC computer onward there was a more or less continuous and self-conscious technical culture of enthusiast programmers, people who built and played with software for fun.

The Real Programmers typically came out of engineering or physics backgrounds. They were often amateur-radio hobbyists. They wore white socks and polyester shirts and ties and thick glasses and coded in machine language and assembler and FORTRAN and half a dozen ancient languages now forgotten.

From the end of World War Two to the early 1970s, in the great days of batch processing and the ``big iron'' mainframes, the Real Programmers were the dominant technical culture in computing. A few pieces of revered hacker folklore date from this era, including various lists of Murphy's Laws and the mock-German ``Blinkenlights'' poster that still graces many computer rooms.

Some people who grew up in the `Real Programmer' culture remained active into the 1990s. Seymour Cray, designer of the Cray line of supercomputers, was among the greatest. He is said once to have toggled an entire operating system of his own design into a computer of his own design through its front-panel switches. In octal. Without an error. And it worked. Real Programmer macho supremo.

The `Real Programmer' culture, though, was heavily associated with batch (and especially batch scientific) computing. It was eventually eclipsed by the rise of interactive computing, the universities, and the networks. These gave birth to another engineering tradition that, eventually, would evolve into today's open-source hacker culture.



3. The Early Hackers

The beginnings of the hacker culture as we know it today can be conveniently dated to 1961, the year MIT acquired the first PDP-1. The Signals and Power committee of MIT's Tech Model Railroad Club adopted the machine as their favorite tech-toy and invented programming tools, slang, and an entire surrounding culture that is still recognizably with us today. These early years have been examined in the first part of Steven Levy's book Hackers [Levy] .

MIT's computer culture seems to have been the first to adopt the term `hacker'. The Tech Model Railroad Club's hackers became the nucleus of MIT's Artificial Intelligence Laboratory, the world's leading center of AI research into the early 1980s. Their influence was spread far wider after 1969, the first year of the ARPANET.

The ARPANET was the first transcontinental, high-speed computer network. It was built by the Defense Department as an experiment in digital communications, but grew to link together hundreds of universities and defense contractors and research laboratories. It enabled researchers everywhere to exchange information with unprecedented speed and flexibility, giving a huge boost to collaborative work and tremendously increasing both the pace and intensity of technological advance.

But the ARPANET did something else as well. Its electronic highways brought together hackers all over the U.S. in a critical mass; instead of remaining in isolated small groups each developing their own ephemeral local cultures, they discovered (or re-invented) themselves as a networked tribe.

The first intentional artifacts of the hacker culture --- the first slang lists, the first satires, the first self-conscious discussions of the hacker ethic --- all propagated on the ARPANET in its early years. In particular, the first version of the Jargon File developed as a cross-net collaboration during 1973-1975. This slang dictionary became one of the culture's defining documents. It was eventually published as "The Hacker's Dictionary" in 1983; that first version is out of print, but a revised and expanded version is New Hacker's Dictionary [Raymond] .

Hackerdom flowered at the universities connected to the net, especially (though not exclusively) in their computer science departments. MIT's AI Lab was first among equals from the late 1960s. But Stanford University's Artificial Intelligence Laboratory (SAIL) and Carnegie-Mellon University (CMU) became nearly as important. All were thriving centers of computer science and AI research. All attracted bright people who contributed great things to the hacker culture, on both the technical and folkloric levels.

To understand what came later, though, we need to take another look at the computers themselves; because the Lab's rise and its eventual fall were both driven by waves of change in computing technology.

Since the days of the PDP-1, hackerdom's fortunes had been woven together with Digital Equipment Corporation's PDP series of minicomputers. DEC pioneered commercial interactive computing and time-sharing operating systems. Because their machines were flexible, powerful, and relatively cheap for the era, lots of universities bought them.

Cheap timesharing was the medium the hacker culture grew in, and for most of its lifespan the ARPANET was primarily a network of DEC machines. The most important of these was the PDP-10, first released in 1967. The 10 remained hackerdom's favorite machine for almost fifteen years; TOPS-10 (DEC's operating system for the machine) and MACRO-10 (its assembler) are still remembered with nostalgic fondness in a great deal of slang and folklore.

MIT, though it used the same PDP-10s as everyone else, took a slightly different path; they rejected DEC's software for the PDP-10 entirely and built their own operating system, the fabled ITS.

ITS stood for `Incompatible Timesharing System' which gives one a pretty good fix on the MIT hackers' attitude. They wanted it their way. Fortunately for all, MIT's people had the intelligence to match their arrogance. ITS, quirky and eccentric and occasionally buggy though it always was, hosted a brilliant series of technical innovations and still arguably holds the record for time-sharing system in longest continuous use.

ITS itself was written in assembler, but many ITS projects were written in the AI language LISP. LISP was far more powerful and flexible than any other language of its day; in fact, it is still a better design than most languages of today, twenty-five years later. LISP freed ITS's hackers to think in unusual and creative ways. It was a major factor in their successes, and remains one of hackerdom's favorite languages.

Many of the ITS culture's technical creations are still alive today; the EMACS program editor is perhaps the best-known. And much of ITS's folklore is still `live' to hackers, as one can see in the Jargon File.

SAIL and CMU weren't asleep, either. Many of the cadre of hackers that grew up around SAIL's PDP-10 later became key figures in the development of the personal computer and today's window/icon/mouse software interfaces. Meanwhile hackers at CMU were doing the work that would lead to the first practical large-scale applications of expert systems and industrial robotics.

Another important node of the culture was XEROX PARC, the famed Palo Alto Research Center. For more than a decade, from the early 1970s into the mid-1980s, PARC yielded an astonishing volume of groundbreaking hardware and software innovations. The modern mice, windows, and icons style of software interface was invented there. So was the laser printer, and the local-area network; and PARC's series of D machines anticipated the powerful personal computers of the 1980s by a decade. Sadly, these prophets were without honor in their own company; so much so that it became a standard joke to describe PARC as a place characterized by developing brilliant ideas for everyone else. Their influence on hackerdom was pervasive.

The ARPANET and the PDP-10 cultures grew in strength and variety throughout the 1970s. The facilities for electronic mailing lists that had been used to foster cooperation among continent-wide special-interest groups were increasingly also used for more social and recreational purposes. DARPA deliberately turned a blind eye to all the technically `unauthorized' activity; it understood that the extra overhead was a small price to pay for attracting an entire generation of bright young people into the computing field.

Perhaps the best-known of the `social' ARPANET mailing lists was the SF-LOVERS list for science-fiction fans; it is still very much alive today, in fact, on the larger `Internet' that ARPANET evolved into. But there were many others, pioneering a style of communication that would later be commercialized by for-profit time-sharing services like CompuServe, GEnie and Prodigy (and later still dominated by AOL).

Your historian first became involved with the hacker culture in 1977 through the early ARPANET and science-fiction fandom. From then onward, I personally witnessed and participated in many of the changes described here.




4. The Rise of Unix

Far from the bright lights of the ARPANET, off in the wilds of New Jersey, something else had been going on since 1969 that would eventually overshadow the PDP-10 tradition. The year of ARPANET's birth was also the year that a Bell Labs hacker named Ken Thompson invented Unix.

Thompson had been involved with the development work on a time-sharing OS called Multics, which shared common ancestry with ITS. Multics was a test-bed for some important ideas about how the complexity of an operating system could be hidden inside it, invisible to the user, and even to most programmers. The idea was to make using Multics from the outside (and programming for it!) much simpler, so that more real work could get done.

Bell Labs pulled out of the project when Multics displayed signs of bloating into an unusable white elephant (the system was later marketed commercially by Honeywell but never became a success). Ken Thompson missed the Multics environment, and began to play at implementing a mixture of its ideas and some of his own on a scavenged DEC PDP-7.

Another hacker named Dennis Ritchie invented a new language called `C' for use under Thompson's embryonic Unix. Like Unix, C was designed to be pleasant, unconstraining, and flexible. Interest in these tools spread at Bell Labs, and they got a boost in 1971 when Thompson & Ritchie won a bid to produce what we'd now call an office-automation system for internal use there. But Thompson & Ritchie had their eye on a bigger prize.

Traditionally, operating systems had been written in tight assembler to extract the absolute highest efficiency possible out of their host machines. Thompson and Ritchie were among the first to realize that hardware and compiler technology had become good enough that an entire operating system could be written in C, and by 1978 the whole environment had been successfully ported to several machines of different types.

This had never been done before, and the implications were enormous. If Unix could present the same face, the same capabilities, on machines of many different types, it could serve as a common software environment for all of them. No longer would users have to pay for complete new designs of software every time a machine went obsolete. Hackers could carry around software toolkits between different machines, rather than having to re-invent the equivalents of fire and the wheel every time.

Besides portability, Unix and C had some other important strengths. Both were constructed from a ``Keep It Simple, Stupid'' philosophy. A programmer could easily hold the entire logical structure of C in his head (unlike most other languages before or since) rather than needing to refer constantly to manuals; and Unix was structured as a flexible toolkit of simple programs designed to combine with each other in useful ways.

The combination proved to be adaptable to a very wide range of computing tasks, including many completely unanticipated by the designers. It spread very rapidly within AT&T, in spite of the lack of any formal support program for it. By 1980 it had spread to a large number of university and research computing sites, and thousands of hackers considered it home.

The workhorse machines of the early Unix culture were the PDP-11 and its descendant, the VAX. But because of Unix's portability, it ran essentially unaltered on a wider range of machines than one could find on the entire ARPANET. And nobody used assembler; C programs were readily portable among all these machines.

Unix even had its own networking, of sorts -- UUCP: low-speed and unreliable, but cheap. Any two Unix machines could exchange point-to-point electronic mail over ordinary phone lines; this capability was built into the system, not an optional extra. In 1980 the first USENET sites began exchanging broadcast news, forming a gigantic distributed bulletin board that would quickly grow bigger than ARPANET. Unix sites began to form a network nation of their own around USENET.

A few Unix sites were on the ARPANET themselves. The PDP-10 and Unix/USENET cultures began to meet and mingle at the edges, but they didn't mix very well at first. The PDP-10 hackers tended to consider the Unix crowd a bunch of upstarts, using tools that looked ridiculously primitive when set against the baroque, lovely complexities of LISP and ITS. ``Stone knives and bearskins!'' they muttered.

And there was yet a third current flowing. The first personal computer had been marketed in 1975; Apple was founded in 1977, and advances came with almost unbelievable rapidity in the years that followed. The potential of microcomputers was clear, and attracted yet another generation of bright young hackers. Their language was BASIC, so primitive that PDP-10 partisans and Unix aficionados both considered it beneath contempt.



5. The End of Elder Days

So matters stood in 1980; three cultures, overlapping at the edges but clustered around very different technologies. The ARPANET/PDP-10 culture, wedded to LISP and MACRO and TOPS-10 and ITS and SAIL. The Unix and C crowd with their PDP-11s and VAXen and pokey telephone connections. And an anarchic horde of early microcomputer enthusiasts bent on taking computer power to the people.

Among these, the ITS culture could still claim pride of place. But stormclouds were gathering over the Lab. The PDP-10 technology ITS depended on was aging, and the Lab itself was split into factions by the first attempts to commercialize artificial intelligence. Some of the Lab's (and SAIL's and CMU's) best were lured away to high-paying jobs at startup companies.

The death blow came in 1983, when DEC cancelled its `Jupiter' followon to the PDP-10 in order to concentrate on the PDP-11 and VAX lines. ITS no longer had a future. Because it wasn't portable, it was more effort than anyone could afford to move ITS to new hardware. The Berkeley variant of Unix running on a VAX became the hacking system par excellence, and anyone with an eye on the future could see that microcomputers were growing in power so rapidly that they were likely to sweep all before them.

It's around this time that Levy wrote Hackers. One of his prime informants was Richard M. Stallman (inventor of EMACS), a leading figure at the Lab and its most fanatical holdout against the commercialization of Lab technology.

Stallman (who is usually known by his initials and login name, RMS) went on to form the Free Software Foundation and dedicate himself to producing high-quality free software. Levy eulogized him as ``the last true hacker'', a description which happily proved incorrect.

Stallman's grandest scheme neatly epitomized the transition hackerdom underwent in the early eighties --- in 1982 he began the construction of an entire clone of Unix, written in C and available for free. His project was known as the GNU (Gnu's Not Unix) operating system, in a kind of recursive acronym. GNU quickly became a major focus for hacker activity. Thus, the spirit and tradition of ITS was preserved as an important part of the newer, Unix and VAX-centered hacker culture.

Indeed, for more than a decade after its founding RMS's Free Software Foundation would largely define the public ideology of the hacker culture, and Stallman himself would be the only credible claimant to leadership of the tribe.

It was also around 1982-83 that microchip and local-area network technology began to have a serious impact on hackerdom. Ethernet and the Motorola 68000 microchip made a potentially potent combination, and several different startups had been formed to build the first generation of what we now call workstations.

In 1982, a group of Unix hackers from Stanford and Berkeley founded Sun Microsystems on the belief that Unix running on relatively inexpensive 68000-based hardware would prove a winning combination for a wide variety of applications. They were right, and their vision set the pattern for an entire industry. While still priced out of reach of most individuals, workstations were cheap for corporations and universities; networks of them (one to a user) rapidly replaced the older VAXes and other timesharing systems.


6. The Proprietary-Unix Era

By 1984, when Ma Bell divested and Unix became a supported AT&T product for the first time, the most important fault line in hackerdom was between a relatively cohesive ``network nation'' centered around the Internet and USENET (and mostly using minicomputer- or workstation-class machines running Unix), and a vast disconnected hinterland of microcomputer enthusiasts.

It was also around this time that serious cracking episodes were first covered in the mainstream press -- and journalists began to misapply the term ``hacker'' to refer to computer vandals, an abuse which sadly continues to this day.

The workstation-class machines built by Sun and others opened up new worlds for hackers. They were built to do high-performance graphics and pass around shared data over a network. During the 1980s hackerdom was preoccupied by the software and tool-building challenges of getting the most use out of these features. Berkeley Unix developed built-in support for the ARPANET protocols, which offered a solution to the networking problems associated with UUCP's slow point-to-point links and encouraged further growth of the Internet.

There were several attempts to tame workstation graphics. The one that prevailed was the X window system, developed at MIT with contributions from hundreds of individuals at dozens of companies. A critical factor in its success was that the X developers were willing to give the sources away for free in accordance with the hacker ethic, and able to distribute them over the Internet. X's victory over proprietary graphics systems (including one offered by Sun itself) was an important harbinger of changes which, a few years later, would profoundly affect Unix itself.

There was a bit of factional spleen still vented occasionally in the ITS/Unix rivalry (mostly from the ex-ITSers' side). But the last ITS machine shut down for good in 1990; the zealots no longer had a place to stand and mostly assimilated to the Unix culture with various degrees of grumbling.

Within networked hackerdom itself, the big rivalry of the 1980s was between fans of Berkeley Unix and the AT&T versions. Occasionally you can still find copies of a poster from that period, showing a cartoony X-wing fighter out of the ``Star Wars'' movies streaking away from an exploding Death Star patterned on the AT&T logo. Berkeley hackers liked to see themselves as rebels against soulless corporate empires. AT&T Unix never caught up with BSD/Sun in the marketplace, but it won the standards wars. By 1990 AT&T and BSD versions were becoming harder to tell apart, having adopted many of each others' innovations.

As the 1990s opened, the workstation technology of the previous decade was beginning to look distinctly threatened by newer, low-cost and high-performance personal computers based on the Intel 386 chip and its descendants. For the first time, individual hackers could afford to have home machines comparable in power and storage capacity to the minicomputers of ten years earlier -- Unix engines capable of supporting a full development environment and talking to the Internet.

The MS-DOS world remained blissfully ignorant of all this. Though those early microcomputer enthusiasts quickly expanded to a population of DOS and Mac hackers orders of magnitude greater than that of the ``network nation'' culture, they never become a self-aware culture themselves. The pace of change was so fast that fifty different technical cultures grew and died as rapidly as mayflies, never achieving quite the stability necessary to develop a common tradition of jargon, folklore and mythic history. The absence of a really pervasive network comparable to UUCP or Internet prevented them from becoming a network nation themselves.

Widespread access to commercial on-line services like CompuServe and GEnie was beginning to take hold, but the fact that non-Unix operating systems don't come bundled with development tools meant that very little source was passed over them. Thus, no tradition of collaborative hacking developed.

The mainstream of hackerdom, (dis)organized around the Internet and by now largely identified with the Unix technical culture, didn't care about the commercial services. They wanted better tools and more Internet, and cheap 32-bit PCs promised to put both in everyone's reach.

But where was the software? Commercial Unixes remained expensive, in the multiple-kilobuck range. In the early 1990s several companies made a go at selling AT&T or BSD Unix ports for PC-class machines. Success was elusive, prices didn't come down much, and (worst of all) you didn't get modifiable and redistributable sources with your operating system. The traditional software-business model wasn't giving hackers what they wanted.

Neither was the Free Software Foundation. The development of HURD, RMS's long-promised free Unix kernel for hackers, got stalled for years and failed to produce anything like a usable kernel until 1996 (though by 1990 FSF supplied almost all the other difficult parts of a Unix-like operating system).

Worse, by the early 1990s it was becoming clear that ten years of effort to commercialize proprietary Unix was ending in failure. Unix's promise of cross-platform portability got lost in bickering among half a dozen proprietary Unix versions. The proprietary-Unix players proved so ponderous, so blind, and so inept at marketing that Microsoft was able to grab away a large part of their market with the shockingly inferior technology of its Windows operating system.

In early 1993, a hostile observer might have had grounds for thinking that the Unix story was almost played out, and with it the fortunes of the hacker tribe. And there was no shortage of hostile observers in the computer trade press, many of whom hads been ritually predicting the imminent death of Unix at six-month intervals ever since the late 1970s.

In those days it was conventional wisdom that the era of individual techno-heroism was over, that the software industry and the nascent Internet would increasingly be dominated by colossi like Microsoft. The first generation of Unix hackers seemed to be getting old and tired (Berkeley's Computer Science Research group ran out of steam and would lose its funding in 1994). It was a depressing time.

Fortunately, there had been things going on out of sight of the trade press, and out of sight even of most hackers, that would produce startlingly positive developments in later 1993 and 1994. Eventually, these would take the culture in a whole new direction and to undreamed-of successes.


7. The Early Free Unixes

Into the gap left by the Free Software Foundation's uncompleted HURD had stepped a Helsinki University student named Linus Torvalds. In 1991 he began developing a free Unix kernel for 386 machines using the Free Software Foundation's toolkit. His initial, rapid success attracted many Internet hackers to help him develop Linux, a full-featured Unix with entirely free and re-distributable sources.

Linux was not without competitors. In 1991, contemporaneously with Linus Torvalds's early experiments, William and Lynne Jolitz were experimentally porting the BSD Unix sources to the 386. Most observers comparing BSD technology with Linus's crude early efforts expected that BSD ports would become the most important free Unixes on the PC.

The most important feature of Linux, however, was not technical but sociological. Until the Linux development, everyone believed that any software as complex as an operating system had to be developed in a carefully coordinated way by a relatively small, tightly-knit group of people. This model was and still is typical of both commercial software and the great freeware cathedrals built by the Free Software Foundation in the 1980s; also of the freeBSD/netBSD/OpenBSD projects that spun off from the Jolitzes' original 386BSD port.

Linux evolved in a completely different way. From nearly the beginning, it was rather casually hacked on by huge numbers of volunteers coordinating only through the Internet. Quality was maintained not by rigid standards or autocracy but by the naively simple strategy of releasing every week and getting feedback from hundreds of users within days, creating a sort of rapid Darwinian selection on the mutations introduced by developers. To the amazement of almost everyone, this worked quite well.

By late 1993 Linux could compete on stability and reliability with many commercial Unixes, and hosted vastly more software. It was even beginning to attract ports of commercial applications software. One indirect effect of this development was to kill off most of the smaller proprietary Unix vendors -- without developers and hackers to sell to, they folded. One of the few survivors, BSDI (Berkeley Systems Design, Incorporated), flourished by offering full sources with its BSD-based Unix and cultivating close ties with the hacker community.

These developments were not much remarked on at the time even within the hacker culture, and not at all outside it. The hacker culture, defying repeated predictions of its demise, was just beginning to remake the commercial-software world in its own image. It would be five more years, however, before this trend became obvious.


8. The Great Web Explosion

The early growth of Linux synergized with another phenomenon: the public discovery of the Internet. The early 1990s also saw the beginnings of a flourishing Internet-provider industry, selling connectivity to the public for a few dollars a month. Following the invention of the World-Wide Web, the Internet's already-rapid growth accelerated to a breakneck pace.

By 1994, the year Berkeley's Unix development group formally shut down, several different free Unix versions (Linux and the descendants of 386BSD) served as the major focal points of hacking activity. Linux was being distributed commercially on CD-ROM and selling like hotcakes. By the end of 1995, major computer companies were beginning to take out glossy advertisements celebrating the Internet-friendliness of their software and hardware!

In the late 1990s the central activities of hackerdom became Linux development and the mainstreaming of the Internet. The World Wide Web has at last made the Internet into a mass medium, and many of the hackers of the 1980s and early 1990s launched Internet Service Providers selling or giving access to the masses.

The mainstreaming of the Internet even brought the hacker culture the beginnings of respectability and political clout. In 1994 and 1995 hacker activism scuppered the Clipper proposal which would have put strong encryption under government control. In 1996 hackers mobilized a broad coalition to defeat the misnamed ``Communications Decency Act'' and prevent censorship of the Internet.

With the CDA victory, we pass out of history into current events. We also pass into a period in which your historian (rather to his own surprise) became an actor rather than just an observer. This narrative will continue in Revenge of the Hackers.


9. Bibliography

[Levy] Levy, Steven; Hackers, Anchor/Doubleday 1984, ISBN 0-385-19195-2.

[Raymond] Raymond, Eric S.; The New Hacker's Dictionary, MIT Press, 3rd edition 1996. ISBN ISBN 0-262-68092-0.

David E. Lundstrom gave us an anecdotal history of the ``Real Programmer'' era in A Few Good Men From UNIVAC, 1987, ISBN-0-262-62075-8.