1 Introduction

In earlier papers of this series (Chavanis 2002a,b,c) we investigated the gravitational instability of finite isothermal spheres in Newtonian gravity and general relativity for classical particles and for quantum particles (fermions). From a theoretical point of view, these systems exhibit interesting behavior with the occurence of phase transitions associated with gravitational collapse (Antonov 1962; Lynden-Bell & Wood 1968; Padmanabhan 1990). A rich stability analysis follows and can be conducted analytically or by using graphical constructions. From an astrophysical point of view, these studies can be relevant for various systems, including elliptical galaxies, globular clusters, the interstellar medium, the core of neutron stars and dark matter made of massive neutrinos.

In this paper, we propose to extend our study to the case of polytropic gas spheres. Polytropes with index 1<n<5 are self-confined, so it is not necessary to introduce an artificial ``box'' to limit their spatial extent. Using the methods developed for isothermal configurations, we show that the transition from stability to instability corresponds to an index n=3. This result is well-known but we provide a new derivation based on the exact resolution of the pulsation equation for polytropes. The perturbation profiles at the point of marginal stability are expressed in terms of the Milne variables (Chandrasekhar 1932). The profile of density perturbation has only one node and the velocity perturbation is proportional to the radial distance.

Then, we consider the case of polytropes with arbitrary index n>1confined within a box of radius R. For $n= \infty$, we recover the classical Antonov (1962) problem for isothermal gas spheres. However, for $n\ge 3$ we already obtain results strikingly similar to those obtained for isothermal configurations. In particular, confined polytropes in hydrostatic equilibrium can exist only below a limiting mass (for a given box radius R) and the series of equilibrium becomes unstable precisely at the point of maximum mass. Subsequent oscillations in the mass-density profile (for n>5) are associated with secondary modes of instability. The locii of these modes of instability follow a geometric progression with a ratio depending on the index of the polytrope. For $n= \infty$, we recover the ratio 10.74... of isothermal gas spheres.

While this paper was in preparation, we came across the preprint of Taruya & Sakagami (2002) on a related subject. These authors investigate the stability of polytropes in the framework of extended thermodynamics, using Tsallis entropy. Therefore, in Sect. 4, we analyze the connexion of the present study with their approach and we discuss the relevance of Tsallis entropy for describing astrophysical systems.