This question has been asked a few times recently on different nail forums. So I'm reprinting a post Bob made a few weeks ago with his suggestions:
If I make a nail photo using a compact camera I try not to use the flash - with the camera so close it over lights the nail and washes it out. Much better to use natural daylight, but preferably indirect light. A cloudy day is the best indirect light - there's no one light source like the sun to create bright spots on the nail or dark shadows. If you're photographing indoors and it's bright outside so several windows cause reflections on the nail - get yourself a sheet and create a tent around the sides of the nail to diffuse the light from the windows.
If you're at an exhibition or training, using the flash is unavoidable. However, before you go, stick a bit of the scotch tape over the flash - it will make the light softer! (use that easy to remove scotch tape not that darn sticky stuff you can't get off your fingers)
Macro is good - but most modern compact camera's have a picture resolution of 8 million pixels or more. What's a pixel - it's a dot of colour used to make up the photo (or show images on your PC screen or tv). 8 million of these blighters is a lot - means you can get good photo quality up to A3 size - bigger than most of us need to put nail photo's on our websites. So you don't have to use macro - simply take the photo and then cut out the nail part of the image to get a closer look. I use Photoshop to edit photographs or re-size them but cheaper software is out there, such as Picasa which is surprisingly good and a free download from Google: http://picasa.google.com/. Note that when you use this non-macro technique, get as close to the nail as the focus on the camera will allow - don't be a million miles away because you will never be able to see the nail close up on print in good quality.
Macro has the advantage it's like a zoom - you can take very close up shots. But the downside is that the depth of focus is less - so the top of the nail might be in focus but the base of the nail may not be (if you don't use macro, the depth of focus is bigger). You need to experiment a bit with your camera to find out what it's limits are here.
Another tip if you use flash is to angle the nail in such a way as the flash doesn't hit the nail head on (90 degrees) - the highlight cause by the flash will be less. Unfortunately so see the nails most clearly that's usually what you have to do. Just move around, take different shots and be lucky!
One final point - before you take the photo, try to think of an interesting background or theme. This will really help to give the photo's "life. Personally, I find this and composing the model the most difficult part - but when it works, it really works! You can look at old nail mags to see get ideas, I also visit jewelry websites for inspiration ;-)
All the photo's below were taken in natural light in a salon. The camera was a cheapo Casio but with 8 Mpixels. No macro was used. We call this one our working girl series - we did it for fun. Iryna is the model and made her own nails. Personally - I like close ups of the nails but also photo's which show the nails in real life situations - I hope Iryna's customers can then better visualise how the nails would look if they had them.