A cooling vest having opposed front and back panels to substantially cover a user's torso. At least one pocket (but preferably a plurality of such pockets) is provided on at least one of the panels and preferably both. The pocket has an opening to receive a cooling pack into the pocket. The pocket further has opposed inner and outer walls normally on the inside and outside, respectively, of the vest. The inner wall has a structural layer and an insulation layer of a different material than the structural layer. Preferably, there are a plurality of such pockets which together cover the majority of at least one of the panels. The insulation layer of the inner wall inhibits too rapid cooling of the user's torso which might otherwise occur from cooling by the gel packs.