We study generalizations of the non-invertible duality defects present in N = 4 SU(N) SYM by studying theories with larger duality groups. We focus on 4d N = 2 theories of class S obtained by the dimensional reduction of the 6d N = (2, 0) theory of A N-1 type on a Riemann surface sigma g without punctures. We discuss their non-invertible duality symmetries and provide two ways to compute their fusion algebra: either using discrete topological manipulations or a 5d TQFT description. We also introduce the concept of "rank" of a non-invertible duality symmetry and show how it can be used to (almost) completely fix the fusion algebra with little computational effort.