1 Introduction

The thermodynamics of self-gravitating systems is a fascinating subject. It started with Antonov (1962)'s discovery that, when a self-gravitating system is confined within a box of radius R, no maximum entropy state can exist below a certain critical energy E=-0.335GM²/R. This intriguing result was further discussed by Lynden-Bell & Wood (1968) who conjectured that for E<-0.335GM²/Rthe system would collapse and overheat. This is called "gravothermal catastrophe'' or "Antonov instability''. Lynden-Bell & Wood have related this phenomenon to the very particular property of self-gravitating systems to possess negative specific heats. The gravothermal catastrophe picture has been confirmed by sophisticated numerical simulations (Larson 1970; Cohn 1980; Lynden-Bell & Eggleton 1980) and is expected to play a crucial role in the evolution of globular clusters. It is found that the collapse proceeds self-similarly (with power law behaviors) and that the central density becomes infinite in a finite time. This instability has been known as "core collapse'' and many globular clusters have probably experienced core collapse (Binney & Tremaine 1987). In the case of dense clusters of compact stars (neutron stars or stellar mass black holes), the gravothermal catastrophe can lead to the formation of supermassive black holes of the right size to explain quasars and AGNs (Shapiro & Teukolsky 1995). Statistical mechanics is also relevant for collisionless self-gravitating systems (e.g., elliptical galaxies, dark matter, ...) undergoing a violent relaxation (Lynden-Bell 1967; Chavanis et al. 1996; Chavanis & Sommeria 1998; Chavanis 1998a, 2001a). In particular, the inner regions of elliptical galaxies are close to isothermal and this is an important ingredient to understand de Vaucouleurs' R^1/4 law (Hjorth & Madsen 1993).

On a theoretical point of view, the stability of isothermal spheres has been first investigated by Katz (1978) with a very powerful method extending Poincaré's theory of linear series of equilibrium. He found that instability sets in precisely at the point of minimum energy. This stability analysis was reconsidered by Padmanabhan (1989) who studied the sign of the second variations of entropy and reduced the problem of stability to an eigenvalue equation. This leads to the same stability limit as Katz but the method of Padmanabhan provides in addition the form of the perturbation that induces instability at the critical point. It is found that this perturbation presents a "core-halo'' structure.

The analysis of Padmanabhan (1989) was performed in the microcanonical ensemble in which the energy is fixed. The microcanonical ensemble is probably the most relevant for studying stellar systems like elliptical galaxies or globular clusters (Binney & Tremaine 1987). Indeed, apart from a slow evaporation, these systems can be assumed isolated so the evolution conserves energy E and mass M. In addition, from the viewpoint of statistical mechanics, only the microcanonical ensemble is rigorously justified for non extensive systems, as discussed in the review of Padmanabhan (1990). However, it is always possible to define formally a canonical ensemble or a grand canonical ensemble in which the temperature is fixed instead of the energy. As suggested by de Vega et al. (1996a, 1996b) these ensembles may be suitable for describing the cold interstellar medium where the temperature is imposed by the cosmic background radiation at $T\sim 3$ K in the outer parts of galaxies, devoid of any star and heating sources (Pfenninger et al. 1994; Pfenninger & Combes 1994). In particular, by working out the statistical mechanics of the self-gravitating gas, de Vega et al. (1996a, 1996b) have shown that self-gravity can provide a dynamical mechanism to produce the fractal structure of the interstellar medium. They used the same approach to explain the fractal structure of the universe (de Vega et al. 1998), assuming that galaxies have reached a quasi-thermodynamical equilibrium like in the early work of Saslaw & Hamilton (1984).

For self-gravitating systems, it is well known that the thermodynamical ensembles do not coincide in the whole range of parameters (Padmanabhan 1990). Using toy models, Lynden-Bell & Lynden-Bell (1977) and Padmanabhan (1990) demonstrated that the region of negative specific heats allowed in the microcanonical ensemble is replaced by a phase transition in the canonical ensemble. This phase transition separates a dilute "gaseous'' phase from a dense "collapsed'' phase. Since these toy models are not very realistic, the self-gravitating gas was also studied in a meanfield approximation. In this viewpoint, an isothermal sphere is stable if and only if it is a local maximum of an appropriate thermodynamical potential (the entropy in the microcanonical ensemble and the free energy in the canonical ensemble). As expected from physical grounds, phase transition occurs when the gaseous sphere ceases to be a local maximum of this potential and becomes a saddle point. In this paper, we investigate the stability of isothermal gaseous spheres in the canonical ensemble by studying the sign of the second variations of the free energy. Our analysis is a direct extension of Padmanabhan (1989)'s approach in the microcanonical ensemble. The two studies therefore provide a unified description of the stability of isothermal spheres in the meanfield approximation in terms of thermodynamical potentials.

For a long time, the thermodynamics of self-gravitating systems was considered exclusively in a meanfield approach (or with toy models). However, recently, de Vega & Sanchez (2001a,b) have developed a rigorous statistical mechanics of self-gravitating systems by using field theoretical methods. In particular, they evidenced the existence of a thermodynamic limit in which the number of particles N and the volume R³ go to infinity keeping N/Rfixed. This very unusual thermodynamic limit proves to be appropriate to non extensive systems. By using Monte Carlo simulations and analytical calculations, they showed that the meanfield approximation correctly describes the thermodynamic limit except near the critical points where a phase transition occurs. They also derived the local equation of state of a self-gravitating gas instead of assuming it, as is done in the meanfield treatments. Therefore, their work fully justifies the studies of previous authors and specifies their range of validity.

This paper is organized as follows. In Sect. 2, we introduce a meanfield description of the system in the canonical ensemble and show that critical points of free energy J at fixed temperature T and mass M correspond to isothermal spheres like those studied in the context of stellar structure (Chandraskhar 1942). We show that there is no global maximum of free energy. There is not even a local maximum of free energy in an unbounded domain: unbounded isothermal spheres have an infinite mass! We restrict therefore our analysis to the case of self-gravitating systems confined within a spherical box of radius R. In this case, there exists local maxima of J (metastable states) if the normalized temperature $\eta={\beta GM\over R}$ is less than 2.52 and the density contrast ${\cal R}=\rho(0)/\rho(R)<32.1$. Critical points of free energy with density contrast ${\cal R}>32.1$ are unstable saddle points. For $\eta>2.52$, there are not even critical points of free energy: in that case, the system is expected to undergo a phase transition and collapse. This "isothermal collapse'' is the counterpart of the "gravothermal catastrophe'' in the microcanonical ensemble (see Figs. 2, 3).

In Sect. 3, we study the sign of the second variations of free energy by using the methods of Padmanabhan (1989) introduced in the microcanonical ensemble. We show analytically that instability sets in precisely at the point of minimum temperature in agreement with the theorem of Katz (1978). The perturbation that induces instability at this point is calculated explicitly; it has not a "core-halo'' structure contrary to what happens in the microcanonical ensemble.

In Sect. 4, we study Jeans type gravitational instability of isothermal gaseous spheres described by Navier-Stokes equations. The introduction of a container removes the problems associated with an infinite homogeneous medium and avoids the Jeans swindle. We show analytically the equivalence between dynamical stability and thermodynamical stability and the fact that the stability of isothermal gas spheres does not depend on the viscosity. This confirms the findings of Semelin et al. (2001) who used numerical methods or approximations. We also give a simpler derivation of the geometric hierarchy of scales inducing instability discovered by these authors using sophisticated renormalization group technics (Semelin et al. 1999). This provides a more illuminating interpretation of their results.

In Sect. 5, we make speculations about the fragmentation and the fractal structure of an isothermal self-gravitating gas. We distinguish between the Jeans length $L_{\rm J}$defined with the mean density and the King's length $L_{\rm K}$ defined with the central density. If we fix the Jeans length, instability occurs for $R>\sqrt{2.52...\over 3}L_{\rm J}$ and is marked by the absence of critical point of free energy (i.e., hydrostatic equilibrium) above this threshhold. In that case the system is expected to collapse without fragmenting. If we fix the King's length (or core size), a first instability occurs for $R={8.99...\over 3}L_{\rm K}$. Above this threshold critical points of free energy still exist but they are unstable saddle points. Secondary instabilities occur at larger box radii that asymptotically follow a geometric progression $R_{n}\sim [10.74...]^{n} L_{\rm K}$. The density profiles that trigger these high order modes of instability are calculated explicitly. They present more and more oscillations whose nodes also follow a geometric progression $r_{n}\sim [10.74...]^{n}R$. The profile that destabilizes the singular isothermal sphere has an infinite number of nodes! Each oscillation can be interpreted as a "germ'' in the langage of phase transition and the above picture suggests that the system will fragment into a series of "clumps''. It is expected that these "clumps'' will evolve by achieving higher and higher density contrasts, and finally fragment in turn into substructures. This yields a hierarchy of structures fitting one into each other in a self-similar way. This picture is given further support by the fact that both the domain sizes inducing instability and the zeros of the perturbation profile in each domain follow a geometric progression with the same ratio. This double-geometric progression may explain in a natural way the fractal structure of a self-gravitating gas like the interstellar medium and the large scale structures of the universe. This gives further support to the interpretation of de Vega et al. (1996a, 1996b, 1998) who emphasized the importance played by self-gravity in building a fractal distribution of matter.