Beyond the conservative dress and showing respect, most required behaviors in Vatican City should be common courtesy, such as: cell phone use (or ringing) is prohibited in churches, smoking is not allowed, and in many areas photos, photos with flash, and/or camera tripods are forbidden, this is especially true in the Vatican Museums. Common curtesy will get you far in Vatican City, but if in doubt ask any employee and they will happily assist you in the rules and etiquette of the country, museums, and churches.
Dress
When it comes to dress in Vatican City conservative is the only appropriate course of action. The city is home to many priests and nuns, both of whom dress very conservatively as the only skin commonly seen is the hands, head, and perhaps the feet, but most people wear closed-toe shoes. The priests tend to wear a more formal outfit at mass, while in public they have more freedom to wear a number of different outfits, but are still somewhat regulated in what can be worn. Many priests also wear a white "collar" so can usually be recognized no matter their clothing. Nuns tend to dress conservatively no matter the situation as some wear a full "habit," while others dress in western styled clothing, but again the dress is very conservative. Even the non-clergy citizens of Vatican City dress conservatively.
One of the most noticeable and famous outfits in Vatican City is that of the Swiss Guard's blue, yellow, and red uniforms. A common rumor is that Michelangelo designed these uniforms, but the reality is that they were designed in the early 1900s and are based on traditional Renaissance uniforms; the present uniforms were likely inspired by painting and frescos from the Renaissance, including many found in Vatican City itself.
For a visitor to Vatican City there are loose restrictions on entering St. Peter's Square, but there is a strict dress code for entering St. Peter's Basilica and this dress code is encouraged for all visitors, even if just entering St. Peter's Square. This dress code forbids hats of any type, does not allow shorts or short skirts (although women may wear skirts that reach past the knees), requires sleeves on all shirts (short sleeves are fine so long as they cover the upper arm), doesn't allow any obscene language, images, or advertisements on clothing, and men are required to wear long pants. These are the minimum requirements to enter St. Peter's Basilica, but many people will dress more conservatively. You are encouraged to wear long-sleeved shirts and pants at all times in Vatican City as a sign of respect.