I am attempting to put together a "Scan probes" guide, based on the information I gathered from a nice developer who took a few minutes of his time to explain them to me and from my own (and other's) experiences. This guide explains how things are in patch 3582. You can treat this information as semi-official.
Probably everyone figured out there are a few different type of probes: those used to scan for a ship (and other things) and those used to scan the moons for minerals.
The first type of probes comes in different flavours that can scan different ranges: 3 AU, 12 AU, 48 AU and 192 AU that can scan respectively 1.5 AU, 6 AU, 12 AU and 96 AU around you (the name of the probe indicates the diameter of the area scanned). What you might have overlooked is that there is a fifth type of probe, Observator Deep-Space Probe; only one of them is needed and can scan the whole system.
There are three more probes used for scanning moons, Quest Survey Probe, Discovery Survey Probe and Gaze Survey Probe. There are quite a few good posts on the forums on how to use these, and their use is pretty straight-forward, so this guide will concentrate only on the other type of probes.
As you have seen on the market interface, there are some blueprints listed for the different types of probes, but none for sale. That is because the developers didn't intend them to be player produced. Alas, you will have to spend quite a lot of money on them, as you will discover soon.
(That applies to both types of scan probes)
There are many guides out there that tell you the same thing over and over, but I didn't find them useful for one particular type of scan, so I decided to write my own. Some of these things are based on my guesswork and heard "through the grapewine", but I will indicate when what I write is the official explanation or the official data (numbers).
(Official explanation) As you may have read in the Player's guide (if you read it), you need to deploy 3 probes in a triangle around your target. The scan area becomes a prism* with a 1 AU up and 1 AU down height (total height of 2 AU). For your probes to yield a result, the target ship has to be somewhere in this scan area.
* The true scan area may actually be the contour of the scan probes' range like in the picture below; I didn't quite get an official explanation to what the area is. I will think of a prism for the sake of simplicity.
(Official explanation) Each scan probe has a "result accuracy" when finding a ship, when this is in its range. It works on a bell curve, for example, if the ship is in point X, the probe will have a result with an accuracy of 20%, if the ship is in point Y, the probe will have a more accurate result, 90%.
Figuring out the position of the ships is done by the ship's computers, so you don't have to be a math wiz for this, but, because of the scan range of the probes, you have to make sure that your target ship is in the range of each probe, otherwise you will not get any results. To find good spots where to drop the probes, you should learn to use the ship's scanner, and there are plenty of good guides out there.
To obtain something useful, you need an average accuracy of at least 50%. For example, you can have one scan probe report a ship with 20% accuracy, and two others report it with 70% and 90% accuracy, your analysis will give good results, because the average is ( 20 + 70+ 90 ) / 3 = 60% accuracy. On the other hand if the scan probes report a ship with 30%, 50% and 60% accuracy, there will be no result from the probe analysis.
Regardless of what probes you use, you have to make sure that each probe will cover the area where your target ship is. Remember that the accuracy is dictated by the proximity of your target ship to the scan probe! Also (I am not sure of this, but I suspect it follows the same logic as the accuracy for the scan probes) you may have to pray that your target ship is closer than 0.5 AU to the plane formed by the three probes in order for it to be found.
Scanning for a ship that is between, let's say, three planets on the same plane is easy and will always give you a result, assuming you used the right probes at the right distances (and the target ship has not moved while you were scanning).
The Observator probe is the only probe that scans in a sphere, and it scans the whole system. You only need to launch one such probe, select it and start analyzing for a ship type of your liking. However...
(Official confirmation) However we are playing a 3D game and using an (almost) 2D scan system. The Observator probe will give you results that are truly three-dimensional. That means you may end up having three (or more) results... if scanning repeatedly for the same ship type... for your target ship location that are unusable, because when you draw a prism between the three points, it does not contain your target ship.
Imagine a sphere around your target ship (with a maximum radius of 96 AU... if it was bigger, the 192 AU probes would not give any results). Take three random points anywhere inside this sphere. Draw a triangle between the two probes and then transform it in a very thin prism. What are the chances your target ship is inside that prism?
I hear that given enough scans and enough probes, you can make use of the results given by the Observator, but it takes a long time and a lot of ISK. For people scanning abandoned ships in space, ships that have no pilots, scanning for those "out of plane" ships will eventually work and you will find it. But if you are in a fleet operation, trying to bust the safe-spot of your opponents, you might want to forget it. By the time you managed, both fleets would have gone to sleep.
I have personally wasted a lot of probes, time, money (both ISK and real, if you remember you pay to play) trying to find ships in space that are not between planets, and repeatedly submitted bug reports about them, but so far nothing has changed. There are rumours that the Kali expansion will change the scanning system to work tri-dimensionally.
(Official data) Some people use only the 3 AU probes, dropping all of them in a location and scanning. This works under certain conditions, and here's the explanation to why. By dropping them all in the same location, you have created a very small triangle (let's say, for the sake of simplicity, it has an area of 1 m2 or smaller). The 3 AU scan probes each scan on a radius of 1.5 AU horizontally (in a circle), and 1 AU vertically. When you perform the analysis all three probes will give results with the same accuracy, and because of that, you have to make sure that the target is under 0.75 AU horizontally and under 0.5 AU vertically. Since this calculation is almost impossible to do with the tools the game offers, you will be certain to have results only if the scan probes are 0.5 AU (or less) away from the target ship: this means you don't have to care about where (in relation to the triangle formed by the probes) your target ship is.
I personally think dropping all three probes and still obtaining a result is a bug, but bugs never get corrected too fast so... might as well use it.
Other people say that you can drop 2 3 AU probes in one spot and another 1 3 AU probe in another spot near the ship and it works. I believe this is a bug as well. If you find that this method doesn't work, try to form a triangle with the probes.
As you see in the analysis screen, you are able to find all the ship types a player can fly. You are also able to scan for "Cosmic signatures" and "Stations". Cosmic signatures are hidden complexes (and I believe mission entrances too?), and analysing for them will reveal the entrance warp gates. Scanning for stations probably reavels only hidden stations (I scanned for stations using the Observator in a system with two listed stations, and I only had one hit.)
Scan probes cannot find cargo containers, drones, NPC ships, agents in space, storyline ships, etc. Basically anything that's not a player ship, a warp gate or a station.
Another limitation of the scan probes is that they cannot find a specific ship out of several ships in the system. Imagine trying to scan for an Ibis in a Caldari starting system... there will be quite a few of them, but how do you know the scan probes just revealed "Donna Darko's Ibis" instead of someone else's? I believe this to be a bug as well, as the scanner interface seems to have the possibility of listing the ship's name. At the moment, there is only the ID of the ship (or the ID of the result of the scan in the database? I have seen this one change occasionally when scanning the same target)... changing that to list the name of the ship should be a piece of cake, but it is again, the player's duty to ask for this to be fixed.
This is all I managed to find out about scan probes. If you know anything else, let me know and I'll add it to this guide.
For reference, CCP measures 1 AU (Astronomical Unit) as 149 598 000 kilometres (which is also the official number).