let coloredBackground = UIView(frame: CGRect(x: 0, y: 0, width: 200, height: 100))
coloredBackground.backgroundColor = UIColor.greenColor()
let imageView = UIImageView(frame: coloredBackground.bounds)
imageView.image = UIImage(named: "myImage")
let label = UILabel(frame: coloredBackground.bounds)
label.text = "stackoverflow"
imageView.maskView = label
The main problem I had was my understanding. Instead of taking a colored view and trying to make a transparent hole in it, we can just layer it the other way around.
So we have the colored background in the back, followed by the image in front that has the mask on it to only show the text part. And actually, that's pretty simple if you're using iOS 8+ by using the maskView property of UIView.