Effect of 13-cis-Retinoic Acid on Serum Prostate-specific Antigen Levels in Patients with Recurrent Prostate Cancer after Radical Prostatectomy¹

The American Cancer Society estimates that 180,400 new cases of prostate cancer will be diagnosed in 2000, of which almost 70% will present with clinically organ-confined disease potentially curable by a radical prostatectomy (1). It is also estimated that 20–50% of the patients subjected to surgery will be found to have locally advanced or metastatic disease after surgery (2, 3). The actuarial nonprogression rate at 10 years after surgery determined by undetectable PSA³ in pathological stage T₁-T₂disease is around 70% (4). Clinical progression of prostate cancer in patients who have had a radical prostatectomy is almost always preceded by a detectable rising serum PSA (5). The time interval between the serum PSA rise and the clinical detection of local or metastatic cancer varies from several months to several years (5, 6, 7). Thus, a rising serum PSA in patients who have had a radical prostatectomy is a harbinger of progressive disease, and these patients are largely incurable by current treatment modalities. Several studies have pointed out the inhibitory effect of retinoids, especially 13-cis-retinoic acid (isotretinoin), on tumor progression and the potential of this class of drugs to promote differentiation of cancer cells (8, 9, 10, 11). Here we report a two-phased study designed to assess the impact of isotretinoin on prostate cancer. In the first phase, we assessed the effect of the drug on the serum PSA level in patients with normal prostates who participated in an oral leukoplakia chemoprevention study. In the second phase, we examined the impact of oral isotretinoin on the serum PSA levels in patients with biochemical recurrent prostate cancer after radical prostatectomy. Any impact of the drug on the serum PSA level in this group of patients can be safely assumed to represent an effect on the cancer because no other prostate tissue is present.

To exclude any effect of isotretinoin on PSA metabolism, we first examined the PSA changes in a control group of patients with normal prostates who were treated with the drug as participants in a chemoprevention clinical trial examining its effect on oral leukoplakia (9). We took advantage of stored, frozen serum samples,which were used to determine PSA levels before treatment and 3 months after treatment with isotretinoin at a dose of 1.5 mg/kg/day (a dose 50% higher than the dose used in our study).

Our targeted study population included males age 40–80 years who had a radical prostatectomy for adenocarcinoma of the prostate within the past 10 years, had rising serum PSA levels between 1.0 and 10.0 ng/ml at the time of enrollment, and agreed to participate in the study. The serum PSA did not have to be undetectable after surgery but had to be persistently rising on at least three occasions within the past year. Metastatic disease was excluded by bone scan, chest X-ray, and abdominal imaging (computed tomography, magnetic resonance imaging, or ultrasound). Prior treatment was acceptable only within the following categories: (a) radiation therapy given before or after the radical prostatectomy; (b) a short course (3–6 months) of androgen ablative therapy given before the radical prostatectomy (neoadjuvant hormonal therapy); and (c)treatment with a 5-α-reductase inhibitor agent, providing treatment stopped before the radical prostatectomy. Patients were excluded from the study if they had received hormonal therapy after surgery, had liver function tests greater than twice the normal range, had serum creatinine greater than 1.5 mg/ml, and had blood triglycerides levels greater than twice the normal range or a cholesterol level greater than 260 mg/ml. Patients were also excluded if they ingested vitamin A orβ-carotene at any level beyond that provided in a standard“one-a-day” vitamin (>25,000 USP units/day) or had taken barbiturates, cimetidine, steroids, or any drug known to affect the metabolism of androgens or of the cytochrome P450 system.

Eligible patients were required to give written informed consent. Twenty patients were programmed to receive oral isotretinoin at a dose of 1.0 mg/kg/day, given as a single daily dose and rounded to the nearest 10 mg, for a period of 12 months. The drug was supplied by Roche Pharmaceuticals in the form of soft gelatin capsules of 10, 20,and 40 mg. The daily dose was established based on effective results and acceptable toxicity described in other tumors (10, 11, 12, 13). The patients were followed monthly for the first 4 months after initiation of treatment and every 3 months thereafter for up to 18 months after the initiation of the study. Serum PSA level was the single parameter of response and was checked at each visit. Pooled PSADT for the treatment group was calculated before starting therapy (at day 0) and at the end of follow-up using the following formula (14): PSADT = log (2) × t/log (final PSA) − log (initial PSA). The time(t) is measured as the midpoint between post-radical prostatectomy nadir and first rise of PSA after nadir until the last PSA test done before initiating therapy (day 0) for pretreatment PSADT and as the midpoint between day 0 and the end of follow-up for posttreatment PSADT.

Serum cholesterol, triglycerides, hematology tests, and blood chemistry were measured at baseline and monitored every 3 months. Serum testosterone and dihydrotestosterone levels were measured at baseline and at month 3 (on therapy) and 15 (off-study) visits. Toxicity was graded according to the Cancer Therapy Evaluation Program common toxicity criteria published by the National Cancer Institute, and a clinical toxicity scale specific for patients treated with vitamin A and derivatives. Patients were taken off study whenever there were signs of clinical detectable disease; a serum PSA rise of >25% per month that did not show any evidence of leveling off by month 6, even without objective evidence of disease progression; grade 3 or 4 toxicity unresponsive to dose reduction; noncompliance; and intercurrent illness necessitating premature termination. Patients going off protocol (completion or discontinuation) were reevaluated for clinical evidence of disease progression by a complete history and physical examination followed by a bone scan, chest X-ray,and abdominal imaging (computed tomography, magnetic resonance imaging,or abdominal ultrasound). The study was approved by the Baylor College of Medicine institutional review board for human studies.

Statistical analysis was performed with SPSS 10.0 (SPSS, Inc., Chicago,IL). The Wilcoxon signed ranks test was used to compare pretherapy PSA levels and PSA levels 3 months after therapy for the treatment and control groups.

Fourteen male subjects were enrolled in the control group (age range,34–68 years; Table 1). Pretreatment serum levels of synthetic isotretinoin were not detected in any patient, and retinol values were in the range of values expected for normal subjects. After 3 months of therapy, high levels of serum isotretinoin and decreased retinol levels were detected in all of the patients but one (patient 12; data not shown). Two patients (patients 10 and 14) had elevated PSA levels of 4.2 and 7.2 ng/ml before treatment. Patient 10 had no history of prostatic disease at the time of our study, but 3 years later, he was diagnosed with benign prostatic hypertrophy. Patient 14 also had no history of prostatic disease and was lost to follow-up. There were no significant changes(P = 1.000) in posttreatment PSA levels (mean, 1.6 ± 1.9) as compared with the pretreatment values (mean, 1.5 ±1.9) in this group (Table 1). For most of the cases, the pre- and posttreatment values were essentially the same (less than ±10%difference for PSA values below 1.0 ng/ml). However, some subjects showed wider variability, including the two subjects who had elevated pretreatment PSA values. These are normal fluctuations commonly observed in noncancerous prostates (15).

Whereas our original goal was to include at least 14 patients, we ended the project after recruiting only 11 patients because, at that point,no patient had a ≥50% decline in the serum PSA, and whereas 3 of 11 patients had some PSA decline, it was of short duration (<6 months). In fact, only one patient (patient 3) completed the 18 months of follow-up, at which point, despite maintaining stable PSA levels (2.0 ng/ml) for several months, he was found to have metastatic disease and was removed from the study. The median age of the patients was 66 years(range, 59–71 years). The pathological stage and grade of the patients’ tumors are described in Table 1. Three patients had a family history of prostate cancer. The mean PSA follow-up period before enrollment into the study (day 0) was 11 months. The mean PSA serum levels at the time of enrollment in the study (day 0) was 2.94 ±1.59 ng/ml. Mean PSA levels after 3 months of isotretinoin treatment increased significantly from 2.94 ± 1.59 to 4.18 ± 2.94 ng/ml (P = 0.023). The pooled PSADT before starting therapy was 6.9 months, and it was shortened to 2.5 months in the period under therapy (mean, 8.8 months), expressing a rapid progression of the disease (Fig. 1). A minimal PSA reduction after initiation of treatment was recorded in only three patients, reducing PSA levels by a maximum of 28%, 15%,and 6.6% for a period of 2–3 months in patients 8, 3, and 11 respectively. In two of these patients, the PSA levels subsequently rose exponentially, and none of these patients completed the 18-month programmed study period. In one patient (patient 10), a stabilization of the PSA curve after an initial sharp rise was seen 2 months after initiation of therapy and sustained for another 3 months before continuing to rise. Overall, a modest and transient effect of isotretinoin on the serum PSA level was seen in 36% of patients. No grade 3 or 4 toxicity was recorded in this group of patients. All of the patients presented grade 1 skin dryness erythema and cheilitis, two patients presented microhematuria (in one case, it was suspected to be due to the use of intraurethral suppository of alprostadil), and two patients complained about arthralgias.

We found evidence of clinical disease progression at the completion of the protocol in only one patient (a biopsy-proven metastatic pelvic lymph node was picked up by the pelvic computed tomography scan).

Retinoids regulate a wide range of biological responses, including cellular proliferation and differentiation, morphogenesis, immune function, and extracellular matrix formation (16). Preclinical studies have shown the retinoids to be effective inhibitors of tumor formation and metastasis in animal tumor models (8). In addition, these compounds have demonstrated clear efficacy in clinical trials involving both chemoprevention of cancer and therapy of established cancer (9, 13).

Retinol and its biologically active metabolite, retinoic acid, were found to be present in normal prostate, benign prostatic hyperplasia,and prostatic carcinoma. However, prostate carcinoma contained five to eight times less retinoic acid than normal prostate or benign prostatic hyperplasia (17).

Retinoids exert their action on tissues through nuclear retinoic acid receptors and retinoid X receptors. Several studies have shown that there is a loss or diminished expression of these receptors in premalignant and malignant tissues, and that the receptors can be up-regulated by retinoid therapy (18, 19, 20). A study examining the effect of 13-cis-retinoic acid on the growth regulation of DU-145 human prostatic cancer cells demonstrated higher mRNA expression for retinoid X receptor α nuclear receptors in cells treated with the drug than in untreated cells. A significant inhibition of growth and metastatic potential of these malignant cells accompanied this up-regulation of receptors (21). The same group of investigators showed that 13-cis-retinoic acid significantly inhibited PSA secretion from the LNCaP human prostate cancer cell line and that the malignant cells became more differentiated, decreasing their growth and tumorigenic potential as compared with controls (22). In athymic nude mice, nearly 50% of the animals showed LNCaP tumor xenograft necrosis followed by complete tumor regression 5 months after treatment with 13-cis-retinoic acid. The combination of androgen ablation and isotretinoin had a synergistic effect (23).

Based on these preclinical data as well as our own retinoid studies (24), we decided to test the effect of oral isotretinoin in prostate cancer patients. We selected patients with a rising serum PSA after radical prostatectomy and reasoned that under these circumstances, the serum PSA can serve as a good intermediate or“surrogate” marker. Evidence from hormonal and radiation therapy suggests that lowering the serum PSA persistently and reproducibly antedates a good clinical response (25, 26). Whether this correlation holds true for other therapies is unknown, and, in fact, we anticipated several possible patterns of PSA response. Retinoids are pleiotropic molecules, and prostate cancer may respond to isotretinoin with either apoptosis or differentiation to a more mature and less malignant phenotype. When this differentiation was observed in vitro in prostate cancer cell lines, it was accompanied by a dramatic enhancement of PSA production (27). Taking these facts into consideration, we anticipated several patterns of response:(a) if the predominant effect were apoptosis, we would have expected a decline in the serum PSA below baseline, similar to the response to androgen deprivation; (b) if differentiation predominated, an abrupt PSA rise (over that expected from the prestudy curve) would be expected in the first month, followed by a plateau (28); and (c) if apoptosis and differentiation occurred simultaneously (in different subpopulations of cancer cells),they may have canceled each other’s effect, showing a stabilization of the serum PSA at baseline levels. In our study of 11 patients, only 3 patients had a minimal decline of serum PSA levels, and in 1 patient,serum PSA levels displayed the pattern of an initial sharp rise followed by a plateau for almost 3 months. Due to these minimal effects attributed to isotretinoin, we did not think it worthwhile to continue this study. Overall, the pooled PSADT calculated for this group under treatment for a mean of 8.8 months was 2.5 months. This correlates with the PSADT profile of patients with distant metastasis, which was calculated to be 5.2 months in another study (29). We felt that these patients would be better served clinically by starting them on conventional hormonal therapy.

Interestingly, DiPaola et al. (12) reported that the administration of isotretinoin in combination with IFN-α did not significantly reduce the serum PSA levels in patients with recurrent disease. These authors reported a 26% partial biochemical response and minimal biochemical response (a 50% and a <50% PSA decrease maintained for 1 month) with a median decrease of 23% in PSA levels at 3 months (12). In another study, Kelly et al. (28) demonstrated that all four patients with rising serum PSA levels after radical prostatectomy or primary radiotherapy treated with a combination of isotretinoin and IFN-2-αresponded within 7–36 days after initiation of treatment by stabilization or decline of the serum PSA level. On the other hand, in the same study, patients with clinically established metastatic disease showed no significant PSA decline, and the limited antitumor effect of retinoids was confirmed by posttherapy tumor biopsies and bone scans (28). Therefore, it is also possible in our study that patients with subclinical metastases failed to respond and that the patients who responded had primarily local recurrences that were not detected clinically.

However, Kelly et al. (26) showed that retinoids are capable of modulating the expression of prostate-specific membrane antigen in prostate cancer toward a more favorable differentiated form of the tumor. This did not translate into a clinically beneficial effect, but it may argue for the exploration of other, more potent retinoids in the treatment of prostate cancer.

Our study is limited by the lack of a randomized placebo control group(although each patient serves as his own control) and by the lack of long term follow-up (i.e., over 1 year). Therefore,we cannot confirm the clinical progression of the disease. Finally, as stated previously, these patients are poorly characterized pathologically. Some may have strictly local recurrence, whereas others may have distant metastases, and still others may have a combination of the two. Clearly, we attempted to alter the course of the disease at a relatively early stage, when the patients suffer from subclinical disease.

Our control group data suggest that isotretinoin has no effect on serum PSA levels in patients with normal prostates. The main purpose of the control group in this study was to exclude a possible direct effect of isotretinoin on PSA secretion or metabolism per se. Consequently, any change in serum PSA level in our study population would reflect a true effect of isotretinoin on the cancer cells. The minimal impact observed is unlikely to translate into a clinical benefit. Nevertheless, these results and the results of the study by Kelly et al. (28) suggest that other retinoid molecules or retinoid combinations should be explored. Additionally, we propose that the two-phased approach used in this study can serve as a model for exploring new pharmacological interventions in prostate cancer, targeting patients with a rising serum PSA after radical prostatectomy in particular.

We thank Roche Pharmaceuticals for supplying the study drug(Accutane) free of charge.