"Chaos" is a tricky thing to define. In fact, it is much easier to list properties that a system described as "chaotic" has rather than to give a precise definition of chaos.
Gleick (1988, p. 306) notes that "No one [of the chaos scientists he interviewed] could quite agree on [a definition of] the word itself," and so instead gives descriptions from a number of practitioners in the field. For example, he quotes Philip Holmes (apparently defining "chaotic") as, "The complicated aperiodic attracting orbits of certain, usually low-dimensional dynamical systems." Similarly, he quotes Bai-Lin Hao describing chaos (roughly) as "a kind of order without periodicity."
It turns out that even textbooks devoted to chaos do not really define the term. For example, Wiggins (1990, p. 437) says, "A dynamical system displaying sensitive dependence on initial conditions on a closed invariant set (which consists of more than one orbit) will be called chaotic." Tabor (1989, p. 34) says, "By a chaotic solution to a deterministic equation we mean a solution whose outcome is very sensitive to initial conditions (i.e., small changes in initial conditions lead to great differences in outcome) and whose evolution through phase space appears to be quite random." Finally, Rasband (1990, p. 1) says, "The very use of the word 'chaos' implies some observation of a system, perhaps through measurement, and that these observations or measurements vary unpredictably. We often say observations are chaotic when there is no discernible regularity or order."
So a simple, if slightly imprecise, way of describing chaos is "chaotic systems are distinguished by sensitive dependence on initial conditions and by having evolution through phase space that appears to be quite random."
In particular, a chaotic dynamical system is generally characterized by
1. Having a dense collection of points with periodic orbits,
2. Being sensitive to the initial condition of the system (so that initially nearby points can evolve quickly into very different states), a property sometimes known as the butterfly effect, and
3. Being topologically transitive.
However, it should be noted that despite its "random" appearance, chaos is a deterministic evolution. In addition, there are chaotic systems that do not have periodic orbits (periodic orbits only survive in the boundaries of KAM tori, and for sufficiently strong perturbations from the integrable case, islands do not necessarily survive). Furthermore, in so-called quantum chaos, trajectories do not diverge exponentially because they are constrained by the fact that the entire evolution must be unitary.
The boundary between regular and chaotic behavior is often characterized by period doubling, followed by quadrupling, etc., although other routes to chaos are also possible (Abarbanel et al. 1993; Hilborn 1994; Strogatz 1994, pp. 363-365).
An example of a simple physical system which displays chaotic behavior is the motion of a magnetic pendulum over a plane containing two or more attractive magnets. The magnet over which the pendulum ultimately comes to rest (due to frictional damping) is highly dependent on the starting position and velocity of the pendulum (Dickau). Another such system is a double pendulum (a pendulum with another pendulum attached to its end).
M. Tabor and F. Calogero have advocated an interpretation of chaos as motion on Riemann surfaces (Tabor and Weiss 1981, Fournier et al. 1988, Bountis et al. 1993, Bountis 1995).
