I'd imagine that you need to HTML encode any literal text you're trying to insert - & would generally be interpreted as the start of an entity, and should normally be encoded as . There's support for HTML encoding in HttpServerUtility.HtmlEncode.
However, we do need to be careful here, since the reported error concerns an XML error, and the list of named entities differs between HTML and XML. I've just been looking in the System.Xml namespace, and can't find any class that specifically helps you produce valid XML data - you might alternatively choose to place your text literals inside a CDATA (<![CDATA[, ]]>) section.