We are studying two new detector technologies that directly measure the three-dimensional coordinates of 511 keV photon interactions for high-resolution positron emission tomography (PET) systems designed for small animal and breast imaging. These detectors are based on (1) lutetium oxyorthosilicate (LSO) scintillation crystal arrays coupled to position-sensitive avalanche photodiodes (PSAPD) and (2) cadmium zinc telluride (CZT). The detectors have excellent measured 511 keV photon energy resolutions (</=12% FWHM for LSO-PSAPD and </=3% for CZT) and good coincidence time resolutions (2 ns FWHM for LSO-PSAPD and 8 ns for CZT). The goal is to incorporate the detectors into systems that will achieve 1 mm(3) spatial resolution ( approximately 1 mm(3), uniform throughout the field of view (FOV)), with excellent contrast resolution as well. In order to realize 1 mm(3) spatial resolution with high signal-to-noise ratio (SNR), it is necessary to significantly boost coincidence photon detection efficiency (referred to as photon sensitivity). To facilitate high photon sensitivity in the proposed PET system designs, the detector arrays are oriented 'edge-on' with respect to incoming 511 keV annihilation photons and arranged to form a compact FOV with detectors very close to, or in contact with, the subject tissues. In this paper, we used Monte Carlo simulation to study various factors that limit the photon sensitivity of a high-resolution PET system dedicated to small animal imaging. To optimize the photon sensitivity, we studied several possible system geometries for a fixed 8 cm transaxial and 8 cm axial FOV. We found that using rectangular-shaped detectors arranged into a cylindrical geometry does not yield the best photon sensitivity. This is due to the fact that forming rectangular-shaped detectors into a ring produces significant wedge-shaped inter-module gaps, through which Compton-scattered photons in the detector can escape. This effect limits the center point source photon sensitivity to <6% for a cylindrical system with rectangular-shaped blocks, 8 cm diameter and 8 cm axial FOV, and a 350-650 keV energy window setting. On the other hand, if the proposed rectangular-shaped detectors are arranged into an 8 x 8 x 8 cm(3) FOV box configuration (four detector panels), there are only four inter-module gaps and the favorable distribution of these gaps yields >8% photon sensitivity for the LSO-PSAPD box configuration and >15% for CZT box geometry, using a 350-650 keV energy window setting. These simulation results compare well with analytical estimations. The trend is different for a clinical whole-body PET system that uses conventional LSO-PMT block detectors with larger crystal elements. Simulations predict roughly the same sensitivity for both box and cylindrical detector configurations. This results from the fact that a large system diameter (>80 cm) results in relatively small inter-module gaps in clinical whole-body PET. In addition, the relatively large block detectors (typically >5 x 5 cm(2) cross-sectional area) and large crystals (>4 x 4 x 20 mm(3)) enable a higher fraction of detector scatter photons to be absorbed compared to a small animal system. However, if the four detector sides (panels) of a box-shaped system geometry are configured to move with respect to each other, to better fit the transaxial FOV to the actual size of the object to be imaged, a significant increase in photon sensitivity is possible. Simulation results predict a 60-100% relative increase of photon sensitivity for the proposed small animal PET box configurations and >60% increase for a clinical whole-body system geometry. Thus, simulation results indicate that for a PET system built from rectangular-shaped detector modules, arranging them into a box-shaped system geometry may help us to significantly boost photon sensitivity for both small animal and clinical PET systems.