This is permissible provided that the shorts are long enough that they cover your awrah; which is the area between the navel and the knees. And that the swimming area is segregated.

Can I go swimming without a shirt and shorts above the knees with my friends or family