Mathematical formulation of quantum mechanics
We have a separable Hilbert space H. The states are the projective rays of H. An operator is a linear map from a dense subspace of H to H. If this operator is continuous, then this map can be uniquely extended to a bounded linear map from H to H. By tradition, observables are identified with operators, although this is rather questionable, especially in the presence of symmetries leading to superselection sectors. This is why some people prefer the density state formulation.
See also Schrödinger picture, Heisenberg picture, Born principle, relative state interpretation.
In this framework, Heisenberg's uncertainty principle becomes a theorem about noncommuting operators. Furthermore, both continuous and discrete observables may be accommodated; in the former case, the Hilbert space is a space of square-integrable wavefunctions.
In the Everett many-worlds interpretation of quantum mechanics, postulate (3) is demoted to a phenomenological principle; see quantum decoherence.
C* formulation
In this formulation, we have a C* algebra, the associative algebra of operators. Positive elements of its dual algebra is are called states and they describe the quantum states. This is related to the density matrix. Given a state, we can construct a unitary representation of it using the Gelfand-Naimark-Segal construction. Two unitarily inequivalent representations are said to belong to different superselection sectors. Relative phases between superselection sectors are not observable.