Hi Bray,
There is a whole zoo of proteins in cell culture medium, especially when serum is added. In addition to the one you know is present and should bind to your antibody, there may be others that are competing for the same binding sites.
What are the specifics of your assay--which antibody are you suing and which protein do you "know" you can detect with said antibody? What's your secondary antibody?
I may be going out a limb here without any of the information above, but I'm strongly suspicious that you are seeing the kinetics of competition amongst several proteins. Kinetics of binding are always concentration dependent, and the fact that you see a peak, plateau, then decrease in your signal suggests means many proteins may be competing for the binding site along with the protein you "know" will bind. That is, you may see a complicated curve because there is an optimal concentration where the kinetics are favorable for your protein to bind. Furthermore, the secondary/detection antibody you are using may be specifically binding to aforementioned protein known to exist, but not the other competitors.
Best advice is to check the specificity of your antibody/protein regime and make sure everything is rational there before proceeding.
good luck!