Given a complex simple Lie algebra $g$ with adjoint group $G$, the space $S(g)$ of polynomials on $\mg$ is isomorphic as a graded $\mg$-module to $(I(g)\otimes \mathscr{H}(g)$ where $I(g) = (S(g))^G$ is the space of $G$-invariant polynomials and $\mathscr{H}(g)$ is the space of $G$-harmonic polynomials. For a representation $V$ of $g$, the generalized exponents of $V$ are given by $\sum_{k \geq 0}dim(Hom_g(V,H_k(g))q^k$. We define an algebra $C_V(g) = Hom_g(End(V),S(g))$ and for the case of $V = g$ we determine the structure of $g$ using a combination of diagrammatic methods and information about representations of the Weyl-group of $g$. We find an almost uniform description of $C_g(g)$ as an $I(g)$-algebra and as an $I(g)$-module and from there determine the generalized exponents of the irreducible components of $End(g)$. The results support conjectures about $(T(g))^G$, the $G$-invariant part of the tensor algebra, and about a relation between generalized exponents and Lusztig's fake degrees.