Be sure to scroll down... there may be more than one question on this page!
For a complete repair the toilet should be removed, all rotten flooring should be completely cut out and replaced with new plywood. Only then can you be sure that you will have a long lasting repair.
But if you only have a "depression" as opposed to an actual hole, you might be able to utilize a floor "leveling compound" such as Level Best from Savogran (online at http://www.savogran.com ) to smooth the surface prior to tiling. The surface must be totally dry and free of dust and loose material.
Multiple coats can be applied... first coat for most of the filling and second coat to bring the repair level with the floor. The first coat should not extend above the surface of the floor at all. Scrape off all excess leveler. The second coat should fill any gaps and bring the repair up the to floor level. If need be, you can apply a third coat if the first two don't result in a smooth-enough floor.
For a small job such as this, you can "rough it" and do the initial and final smoothing with a dampened straight board. A pro would use a magnesium float... a concrete finishing tool that can give a wonderfully smooth surface... but I am hoping to save you the added expense of this tool.
Once set and totally dry, Level Best can be sanded by hand or with a power sander. It will sand more easily with power equipment such as a belt sander or grinder, but they raise large amounts of dust that should not be inhaled. Use a vacuum to catch as much dust as possible, provide for ventilation and (at the least) wear a good fitting dust mask. Once the repair is totally dry, smooth and dust-free, you should be able to put the tile right over it without the need for a primer.
Level Best is not designed to be used in wet locations, so if you decide to use it you should take care to keep the floor dry from future drips. There are other leveling products that are cement-based (Level-Best is similar to plaster) but have a less forgiving finish and cannot be easily sanded, making them more difficult to use. However, these cement-based products stand up to moisture better and may be a superior choice if you are willing to take more care with the finishing.
There are two potential problems in installing self-stick tiles over a subfloor. One is the smoothness of the subfloor. To be blunt, the floor should be absolutely smooth! The reason is because any irregularities.. lumps, bumps, seams, nail holes, etc... in the subfloor surface will be transferred through to the tiles over time. This is true of vinyl tile installation over any surface. Even the pattern of a textured sheet vinyl or linoleum floor will appear through the self-stick tile! Now if you are only talking about a imperfections or nail-heads, these can be easily repaired by patching with a quality wood filler or floor leveling compound. Don't use wallboard patching materials because they may not be hard enough. Sand surface smooth, vacuum and damp-clean to remove all dust before tile installation.
The second problem is at the seams of the floor. If there is any "flexing" or movement at the seams, the tiles will eventually crack across these seams. In most situations, the easiest way to firm up the floor is to install a layer of 1/4" plywood over the subfloor (nailed or screwed and glued) in such a way that the new plywood seams do not lay over the old seams. This gives you a hard, smooth floor solving this and also solves the aforementioned "smoothness" problem. The nail or screw heads should be "set" below the surface of the floor and the holes filled.
I think the pro was wise to suggest the additional plywood, since he is well aware of the problems associated with poor subfloor preparation. He also knows that, time wise (since his time is money), it could very well take longer to repair a bad floor than to just cover it with plywood. This may not be true in the case of a do-it-yourselfer such as yourself, since you are looking to save money, not time!
In my opinion, the first part of this job should be to remove the existing warped subfloor and replace it with fresh plywood. Vinyl tiles require a very smooth and solid surface. The installer of the original subfloor probably did not both glue and screw it down. The plywood lifted due to normal or even abnormal movement in the older floor underneath, pulling some nails in the process I am sure. To remove the old floor, you can try to pry up the plywood from the edges or take a few more minutes and remove each nail (or screw) individually. I let the conditions of the job determine the method… if the plywood breaks apart it is sometimes easier to remove the nails and remove each panel whole. If the plywood is rigid and can be pried up without breaking, try to bring up the sheets. Of course, you can also make cuts through the subfloor to break the sheets up into more manageable pieces. Just don't cut through the plank flooring!
Now, there is the question of the original wood plank subfloor. If there is any movement in this floor, © inch plywood will not stabilize it. So your next job is to nail or, preferably, screw down the old floor as necessary so that it is absolutely solid and squeak free. Then install the new plywood being sure to end sheets between the seams in the old floor. Construction adhesive can be used between the old floor and the new plywood to add additional strength and stability to your "floor sandwich". Yum.
It is generally not necessary to fill small spaces between the pieces of plywood. However, if the gaps approach © inch or if there are depressions or defects in the plywood face such as deep knots, they should be filled. These wider gaps can be weak spots that may cause breakage in the vinyl tiles if there is direct pressure applied to them, such as from a table leg or a high heel. Leveling compounds are powders that are mixed with water and handle somewhat like wall plaster. They can be poured into these cracks or troweled into the depressions. Follow the package instructions carefully and be sure to trowel the repair as flat as possible to minimize sanding. These products dry very hard, don't sand easily and tend to load up (stick to) the sandpaper.