Validation of Selected, Non-Commercial (Q)SAR Models for Estrogen Receptor and Androgen Receptor Binding

Transcription

1 Validation of Selected, Non-Commercial (Q)SAR s for Estrogen Receptor and Androgen Receptor Binding Sponsor: European Commission Directorate General Joint Research Centre Institute for Health and Consumer Protection Service contract n. CCR.IHCP.C X0 Contractor: Istituto di Ricerche Farmacologiche Mario Negri, Milano, Italy Authors: Emilio Benfenati, Alessandra Roncaglioni, Elena Boriani, Chiara Porcelli, Morena Spreafico & Elena Lo Piparo FINAL REPORT 12 December 2005

2 Disclaimer by the Joint Research Centre Any conclusions and opinions expressed in the report belong to the authors, and do not necessarily represent the views of Commission staff. The European Commission accepts no liability for the use of any information in this report. Andrew Worth European Commission - Joint Research Centre Institute for Health & Consumer Protection

3 Table of contents 0 SCOPE OF THE REVIEW ORIGINAL PROJECT PLAN RESULTS...5 WP1: Gathering s Library search on scientific publications Search through EC funded projects Search in the World Wide Web Search through a specific questionnaire Other sources (BioByte QSAR database)...9 WP2: Criteria for Short Listing of (Q)SAR Selection...10 Preliminary classification of the models for AR and ER binding and tranactivation...10 Criteria for model review...10 WP3: Application of Short-Listing Criteria for the Final (Q)SARs Selection...13 WP4: Additional analyses for the sort-listed (Q)SARs : ER Relative Binding Affinity (RBA) for rat...18 a. Activity data...18 b. Structures and chemical descriptors...18 c. Internal validation: model and model s performances...18 d. External validation...20 e. Applicability domain a: SAR model for ER Relative Binding Affinity (RBA) for rat...33 a. Activity data...33 b. Structures and chemical descriptors...33 c. Internal validation: model and model s performances...33 d. External validation...35 e. Applicability domain b: Decision forest model for ER Relative Binding Affinity (RBA) for rat...38 a. Activity data...38 b. Structures and chemical descriptors...38 c. Internal validation: model and model s performances...38 d. External validation...38 e. Applicability domain : AR Relative Binding Affinity (RBA) for rat : ER Reporter Gene Assay...43 WP5: Proposal for the development and validation of new, high quality (Q)SARs CONCLUSIONS...46 References...47 Annex A...50 Annex B...60 Annex C...77

4

5 0 SCOPE OF THE REVIEW Recently an increasing concern arose from the scientific community [1, 2] and political institutions [3] on chemical contaminants which can interfere with endocrine system, the so called endocrine disrupters. As a consequence it was recognise the importance to address further research in this direction [4]. These compounds in most of the cases appear to act through the binding to Nuclear Receptors (NRs). Nuclear receptors are a superfamily of ligand-dependent transcription factors that mediate the effects of hormones and other endogenous ligands to regulate the expression of specific genes [2]. Members of the NR superfamily include receptors for various steroid hormones -estrogen (ER), androgen (AR), progesterone (PR), and several corticosteroids-, retinoic acid, thyroid hormones, vitamin D, and dietary lipids (the peroxisome proliferator activated receptor: PPAR). A large number of orphans NRs have also been identified whose cognate ligands are still unknown [5]. Diminished or excessive production of a particular hormone or target-cell insensitivity to a hormone is among the major problems related to human endocrine dysfunction diseases. Two general lines of study have been explored in respect of NR binding models using in silico tools: 1) Much of the current understanding of NR molecular and cellular biology and pharmacology is a result of decades of studies in the pharmaceutical research. The main target of this kind of studies is to identify candidate drugs for hormone-related disease (e.g. Selective Estrogen Receptor Modulators as treatment of hormone-dependent tumours). In this kind of modelling usually a small quite homogeneous group of compounds is taken into account and the highest grade of mechanistic information is included. This implies that not only the chemical itself but also the biological macromolecule is considered in the analyses (like in docking studies). Reports of the crystal structure with bound agonists and antagonists [6, 7] have advanced the understanding of critical features of the ER ligand binding domain and allowed for further exploration of the roles of activation factor regions of the protein in binding and coactivator interactions. Site-directed mutagenesis techniques have also allowed study of the contributions of single receptor residues or sequences critical to NR binding (see [8] for ER). 2) In addition to being of interest from a pharmaceutical perspective, there is considerable concern regarding the potential for adverse effects in humans and wildlife from environmental exposures to chemicals that can potentially interact with and disrupt any of a number of endogenous hormone pathways (e.g., estrogen, androgen, thyroid). Studies dealing with endocrine disrupters have for some extent a different perspective and use generally slightly different tools; furthermore, the datasets are usually wider. The final goal of this kind of study is also quite different compared to that exposed in point 1) above, since it is related to the screening of (potential) environmental contaminants for their possible interfering with endocrine system with the aim to identify all potential EDs: the target is to avoid the so-called false negatives. Vice versa, pharmaceutical industries in their HTS studies wants to focus on the most likely active compounds, avoiding false positive. From this point of view the use of ER/AR binding test could be not sufficient because the down streams events to the binding can produce quite different effects if the compound behaviour like an agonists or antagonists; consequently the relevance of the possible disruption effects will depends on the transcriptional activation of DNA. Over the past decade, a great deal of interest has been put in the development of QSAR models and methodologies. The range of activity explored by these models range from physico-chemical characteristics, to carcinogenicity. The use of QSAR, applied to endocrine disruptors, is quite widespread [9-11] and the U.S. Environmental Protection Agency too was involved in the endocrine disruptors screening programme [12], with efforts in the development of models to predict receptormediated response [13, 14]. 2

6 The docking method was widely applied to various NR. It helped in understanding the mechanism of action of many chemicals and it is a powerful screening method for new potential endocrine disruptors. In some studies the computed docking binding energy is use for development of QSARs and qualitatively define the mechanism of action [15-17]. Furthermore, the available crystal structures of NR allowed the building of homology models of other sub member s proteins of this big family [18]. The helpfulness of this approach was shown in several papers where different NRs were compared and their mode of ligand-receptor interaction was found. One of the most widely available and often-employed receptor- independent 3-D QSAR approach utilizes comparative molecular field analysis (CoMFA) [19]. CoMFA helps also to highlight the differences in the NR sub types and modelling the selectivity of the ligands to some NR. Tong et al. [20] employed CoMFA maps to identify and differentiate the structural features of estrogens responsible for ligand binding to ER-alpha and ER-beta. Even when the receptor crystal structure is available, CoMFA provides an additional source of information about the receptor from the perspective of the ligands. It should be noted that the crystal structure of a ligand receptor complex provides a single, low-energy, snapshot of the actual dynamic biological system. The alignment of the compounds is a critical step for CoMFA. To more precisely identify ligand-protein residues interaction and define the mode of alignment, a structure-based 3D-QSAR [21] approach was applied to some NR types. This technique uses the docking conformers in the CoMFA alignment and makes a bridge between the two methods. CoMFA, HQSAR and CODESSA were used in combination providing a helpful comparison of their predictive power [23]. The results show significant advances of the 3D techniques compared to 2D, probably related to the descriptors used. From this point of view, there is an increase of studies using new 2D descriptors and mathematical approaches like for example Volfsurf [23] and holographic QSAR [10, 22]. The present report contains the results of a detailed literature review conducted in the framework of the present project in order to: i) Identify literature sources describing relevant modelling exercises for endocrine disruptors in terms of oestrogen or androgen receptor interferences, more precisely referring to binding affinity and transcriptional activity. ii) Organize the collected the material in terms of complete analysis of the available models and techniques adopted to model the selected end-points. iii) Select the most promising models and evaluate the feasibility to reproduce them. 3

7 1 ORIGINAL PROJECT PLAN The project plan consisted of five major tasks, listed in Table 1. To each task the corresponding work package (WP) has been assigned. Table 1. Work packages planning table. Tasks: WP Gathering models. 1 Criteria for short-listing of (Q)SARs selection. 2 Application of short-listing criteria for the final (Q)SARs selection. 3 Additional analyses for the short-listed (Q)SARs. 4 Proposal for developing a high-quality QSAR. 5 The following chapters describe in details the progress obtained in all work packages. 4

8 2 RESULTS WP1: Gathering s We reviewed non-proprietary models for ER and AR binding activity and for the prediction of transcriptional activation. For this purpose, a search of data has be done in the following ways: 1. through a library search on scientific publications 2. through results from EC funded projects 3. through the internet, with particular attention for specific website for nuclear receptors activity 4. search through a specific questionnaire 5. other sources (BioByte QSAR database) 1. Library search on scientific publications In the literature the number of QSAR models for estrogens, androgen receptors binding activity are growing. We made a library search on scientific journals, scientific publications in general and more in particular on journal referred to QSAR, toxicology and those referred to endocrine system using specific keywords. Thus a focussed search in the literature has been done: a. On models for ER, AR binding b. On models for the prediction of transcriptional activation c. On ER and AR nuclear receptors and libraries of compounds d. On authors which have worked in the field, for further check The list of articles collected in this way is reported in the Annex A and contains more than 150 publications. All publications are here quoted using a reference number increasing for the most recent publications (except for a few - Ref. Nr added in a later stage). An initial analysis of these papers allowed us to classify them in some categories providing a preliminary overview of what is available in literature. Most of the models investigated preferably ER compared to AR end-points (see Figure 1); this is probably due to the interest on ER for pharmaceutical targets, an because of an early development and standardization of the test procedures for ER that allowed to obtain a bigger number of data to be modelled Estrogen Receptor Androgen Receptor Figure 1. Number of articles dealing respectively with ER and AR end-points. In Figure 2 the relative proportion of end-points studied in the collected papers is shown. The greater part of them investigated the binding affinity as preferred end-point. Beside the binding, the other end-point of interest for this project is related to the transcriptional activation. This type of study is interesting because the effect of compounds which binds to nuclear receptors is different if they behave as agonist or antagonist. 5

9 Figure 2. Number of articles dealing with the different end-points Binding Assay Reporter Gene Assay Cell Proliferation Assay In vivo Finally a fist classification was attempted to detect the more popular techniques used to model in the papers in 4 categories (see Figure 3): Docking. The docking approach consist in the prediction of the binding modes between small molecule ligands and large protein targets using in silico tools. Molecular docking is often used in virtual screening methods, whereby large virtual libraries of compounds are reduced in size to a manageable subset, which, if successful, includes molecules with high binding affinities to a target receptor. 3D-QSAR. Perhaps the most widely available and often-employed receptor-independent 3-D QSAR approach utilizes CoMFA [19]. The database of molecules with known properties are suitably aligned in 3D space according to various methodologies of superimposition. On the basis of the chosen alignment, charges are then calculated for each molecule. Steric and electrostatic fields are computed for each molecule by interaction with a probe atom at a series of grid points surrounding the aligned database in 3D space. The model then attempts to correlate these field energy terms with a property of interest by the use of partial least squares (PLS) with cross-validation, giving a measure of the predictive power of the model. Conventional QSAR. Conventional QSAR analysis differs from 3D-QSAR since molecular descriptors are used instead of interaction fields. Molecular descriptors can require a geometry optimisation to calculate them on the basis of the more reasonably energetic conformer (or pool of conformers). Other descriptors are insensitive to the geometry shape of the molecules and thus do not require an optimisation step. SAR. A more qualitative analysis of the data It is quite obvious to note that in this case, where the target is well known and the receptor solved at the crystal structure level the studies which involves docking and 3D-QSAR are more common than in case of studies focused on other endpoints, for instance aquatic toxicity Figure 3. Number of articles dealing with the different techniques SAR QSAR 3D-QSAR Docking 6

10 2. Search through EC funded projects EC funded projects represent an excellent and easily accessible source of information on ongoing research. The knowledge so far acquired through them provides a reliable starting point in collecting models candidates. There are/have been some EC projects which are studying/have studied QSAR models for endocrine disruptors; several other projects have been evaluated in particular to detect valuable sources of data issue since there was a special call in the FP5 (QoL/ENV-2001-ENDO) dedicated to endocrine disrupters. Below a brief outline of some important project dealing with QSARs and endocrine disruption: EDAEP, Endocrine Disrupting Ability of Environmental Pollutants, ENV4-CT Objectives of EDAEP were: i) Develop and apply QSAR (Quantitative Structure Activity Relationship) techniques for the screening of HPVCs (High Production Volume ), based on existing experimental data and improve models using data arising from in vitro and in vivo assays on selected chemical. ii) Develop and implement an in vitro and in vivo testing strategy for EDCs. iii) Establish a priority list of chemicals to validate QSAR approach. Results and conclusions: The QSAR technique was applied to screen 908 HPCV (high production volume) chemicals for their conformational affinity to human oestrogen receptor alpha. A set of 40 HPV showing Relative Binding Affinity indexes ranging from 1 to 10%, 10 to 100% and >100%, with respect to oestradiol, was selected for in vivo and in vitro testing. Out of the 908 HPVCs tested, the QSAR model screening did not indicate compounds with RBA>100%. Of the forty chemicals selected for testing only bisphenola-diglucylether had a RBA % and was found positive at the HER-binding, YES and CarpHEP/Vtg assays, while the negative response obtained by the E-Screen and in vivo assays on fish Vtg and uterotropic assays in female rats was ascribed to metabolic deactivation of the parent compounds. Similarly, the positive response obtained by the E-Screen and uterotropic assays on Tetrabromo-bisphenolA is supported by metabolic de-bromination cleavage in vivo, producing BisphenolA. Of the five HPVCs with RBA 1-10%, only BisphenolA gave a positive response with all the assays applied. Based on these results, the QSAR COREPA model was further refined and improved to test Low Production Volume (LPVCs). Nine LPVCS selected from the list of QSAR predicting 100% RBA were further tested by in vitro and in vivo assays, giving positive responses by ERE-CALUX, herbinding assays, E-Screen and (5 on 8) uterotropic assay. By further refinement and optimisation, the QSAR Corepa model could become a useful tool for reducing uncertainty in screening large chemicals inventory. This development should relate to validation on a more homogenous set of testing methods in terms of sensitive end-points and species. Several articles by Mekenyan et al. were found in literature exploiting results from EDAEP. CASCADE, as contaminants in the food chain: an NoE for research, risk assessment and education, FOOD-CT The project CASCADE is studying endocrine disruptors and their role in food; this research includes QSAR models on endocrine disruptors. A questionnaire for this purpose has been distributed. The CASCADE Network of Excellence, launched in February 2004 within the EU's Sixth Framework Programme, aims to provide Europeans with a durable, comprehensive and independent network of excellence in research, risk assessment, and education on health risks that are associated with exposure to chemical residues in food. Focus lies on chemicals that act via and/or interfere with cellular regulation by hormone receptors. CASCADE's mission is not only to fill gaps of scientific knowledge, but also to increase knowledge among the general public about research on food-borne chemicals. Activities include state-of-the-art research from the best European research centres in the field, training programmes for students, relevant information to consumers and reliable assessment to legislators concerning risks of chemical contaminants in food. In the present project we used the outcome from the questionnaire circulated within CASCADE. Docking studies have been done on many nuclear receptors, screening about 500 compounds from a list of contaminants of European interest, to identify likely active compounds. 7

11 EASYRING, Environmental Agent Susceptibility Assessment utilising existing and novel biomarkers as Rapid non-invasive testing methods, QLRT Within EASYRING a specific WP is dedicated to the development of a database regarding the ability of chemicals to elicit endocrine disruption and to utilise these publicly available data to develop quantitative structure-activity relationship (QSAR) models with a peculiar attention to the extrapolation of effects to humans. Some of the results are already published in literature and personal communications with the WP leader (Dr. Cronin) as participant in the project have been established. FIRE, Risk assessment of brominated flame retardants as suspected endocrine-disrupters for human and wildlife health, QLK4-CT This project is focused on brominated flame retardants (BFRs), such as the high production volume chemicals polybrominated diphenyl ethers (PBDEs), tetrabromobisphenol A (TBBPA) and hexabromocyclododecane (HBCDD). For the prescreening of BFRs a battery of in vitro assays and QSAR models are used, to: i) determine the endocrine disrupting potency; ii) to select the test compounds for testing in in vivo studies; and iii) to determine the appropriate test concentrations for in vivo studies. Marie Curie host fellowship, Chemometrical Treatment of Toxic Compounds Endocrine disrupters, HPMT-CT This project focus on the development of (Q)SARs models from a qualitative and quantitative point of view by using artificial neural networks (ANN). Some of the results are already published in literature and personal communications with the project leader (Dr. Novic) have been established. 3. Search in the World Wide Web We made a search within specific web portal listed below: Among them the last four were definitively the most useful since they contains some datasets used in QSAR or that can be useful in the project. In some of these websites references to published models and structures behind them are given (models by Tong et al. on ER and AR binding). 8

12 4. Search through a specific questionnaire We used results of a questionnaire which have been done promoted by the EC, within the EC project CASCADE on DATABASE and QSARs. The questionnaire has been sent by us to all the co-ordinators of EC projects on endocrine disruptors and a wide number of scientists (> 160 groups) in Europe and outside. 24 replies were received and only few of them quoted QSAR studies; references from literature were given. 5. Other sources (BioByte QSAR database) QSAR equations based on the classical Hansch approach were collected from the BioByte QSAR database. For estrogens more than 100 equations are stored in the database using a number of compounds ranging from 4 to 49. For androgens 5 equations are stored in the database using a number of compounds ranging from 6 to 11. Several details are given on these models but due to the congeneric series and the low number of compounds used, these models are of a limited utility. 9

13 WP2: Criteria for Short Listing of (Q)SAR Selection Preliminary classification of the models for AR and ER binding and tranactivation To better structure the review procedure, we summarized the articles. Simple key points were defined for such a purpose. This analysis is intended to provide essential information about the QSAR under investigation, facilitating comparison on similar models, developed by the same authors investigating similar datasets. More detailed evaluation criteria will be considered in a subsequent evaluation step. Table 2 shows a preliminary list of key points used to summarize and classify the models. Table 2. Preliminary key points analysed for classifying QSAR models for ER and AR binding and transactivation activity. Keyword Endpoint Chemical Domain Chemical structure and descriptors Statistics Description Preliminary definition of the assay, tested species and test method. The chemicals the QSAR can be applied to. Information about the structure preparation and about the kind of chemical description used information about the mathematical model used for modelling, including software Statistical parameters given to define de goodness-of-fit and predictivity of the model, including the test set After a more detailed evaluation of the collected papers it appears that some of them were of little interest for this specific project for several reasons and where then excluded from more detailed analysis. Indeed, in some cases the papers contained only very generic description of the approaches adopted or review based on other published works. The description shown in Table 3 was then applied to the remaining articles and reported in the Annex B of this report. Criteria for model review A preliminary evaluations of the OECD principles Since the main goal of this project is related to the validation of model(s) for their compliance with the OECD principles on the basis of the preliminary analysis of the available papers some issues are here summarized: Regarding the first principle it is interesting to note that there is a high interest in endocrine disrupters from a regulatory point of view but validated tests and guidelines do not already exists. We cannot solve this problem within the framework of this project so the strategy adopted was to ensure as much as possible that the data used to derive the QSAR were of a good quality. Later on when standardized guidelines will exist it will be possible to verify if the data used do derive the selected and validated QSAR are in agreement with those coming from new standardized protocols. About the second principle and the explicit definition of the model and descriptors many problem may arise (especially for 3D- QSAR models, strongly depending upon the alignment) since further tests are needed to verify if the model is reproducible using available information and eventually contacting directly the authors of the most promising QSARs. The third principle, dealing with the definition of the applicability domain, will not be used for ranking the articles since this information is almost never available a priori in the original work. A more interesting point that is somehow related to the third principle and will be carefully evaluated to select the preferred models, is related to its utility in terms of endocrine disruptors screening. This is a very crucial point since many models were developed using small datasets of very similar synthetic candidate drugs and these models will be dubiously useful for a large screening of industrial chemicals. 10

14 Also the fourth principle, dealing with the goodness-of-fit and predictivity of the models, a precise task of the present project will be exactly to independently assess the quality of the validated models. Furthermore, since all models are very heterogeneous and adopted ad hoc parameters to assess their performances, this piece of information will be used only marginally. The fifth principle about the mechanistic interpretation will not be considered to select the most preferable QSAR models because this point is well addressed in the papers. In conclusions there are some critical points since many OECD principles are not satisfied directly within the original papers. Definition of the criteria suitable for short-listing of QSAR. The preliminary scoring of all models was modified from the original one proposed in the tender on the basis of the analysis of the available papers and in view of the useful discussions at the first meeting of the project and already anticipated in the previous paragraph. The criteria and the assigned scores are listed in Table 3 and rationalized below. Table 3. Definition of criteria and scores. Index Criteria name Score a Experimental biological data 0-1* b Chemical information: structures 0-1* c Chemical information: descriptors 0-2* d Chemical domain 0-2 e ling approach and feasibility of the model 0-3 * These criteria have necessarily not to be 0 to proceed in the evaluation. If 0 the paper is not considered. a. Experimental biological data should be reported for each chemicals in the exact terms of the modelled output or detailing the necessary transformation (range of activity for defining classes, logarithmic transformation of the continuous values). Data can be reported in another source but should be available. This will be a minimum requirement to consider the model for the next steps. Due to the short time available for the project it was problematic to contact directly all authors to ask for this piece of information. b. Chemical information on the structures should be given; at least in terms of sketched 2D structure or chemical name to identify the substances under evaluation in order to be able to derive the data matrix associating chemicals to the activity data. This will be a minimum requirement to consider the model for the next steps. c. Chemical information on the descriptors and how they were calculated should be given. If the calculated descriptors and in particular the descriptors used to build up the model are only generally defined (e.g.: reactivity parameters without any further detail) they can not be recalculated and so score 0 will be assigned to these models. A discrete scale of this score is necessary to take into account: i) the easiness for reproducing the descriptors (e.g.: CoMFA descriptors depending on the alignment) and ii) if the table with the calculated descriptors is given, in order to check for consistency with the new recalculated values. d. A specific score will be assigned to the chemical domain (different from the domain of applicability). Even in case of formal acceptability, utility of the models can be very limited because very focused on the chemical space. This score will somehow encode for the real utility of the developed model for screening of industrial chemicals, based on the qualitative assessment of the chemical diversity and for the interest in the studied chemical classes from an environmental and toxicological point of view 11

15 e. This score refers to several points connected with the modelling task. i) Basically it encodes for the availability of sufficient information on the parameters of the model which allow to reproduce it. ii) A further point relates to the feasibility of the modelling approach. This means that easier models are privileged with an higher score but also more complex models can be selected as well. iii) Finally an overall evaluation of the models is given in terms of its performances, simply to avoid the selection of models which already demonstrate to be relatively weak in their performances of failed some validation attempts in the original paper. For each model a total score will be estimated by adding the individual scores gained within each criterion, but only if the first 3 criteria are all different from 0. This procedure provides an objective basis for the comparison among candidate models. 12

16 WP3: Application of Short-Listing Criteria for the Final (Q)SARs Selection The above mentioned selection criteria for QSAR short-listing will be applied to evaluate all non commercial QSARs collected during the initial stage of the project. Application of selection criteria would allow us to: 1. provide a comprehensive set of scored models for Androgen and Estogen binding, and for Transcriptional Activity prediction. 2. provide criteria for comprehensive, appropriate, referenced, quality-controlled data for all model inputs. 3. include appropriate methods for taking account of uncertainty, and variability. Table 4 summarizes the scores for all collected articles. Many of the papers did not contain useful models, described proprietary models or were review papers not reporting original models. These articles where not taken into account and specific scores were not assigned. The same approach was applied to articles that did not pass the first 3 criteria (a, b and c). All these articles were not ranked but discarded and inserted at the end of the table. Table 4. Application of the scores. Ref. Nr. Notes End-point Exp. data a. b. c. d. e. Structure info Descriptor Chemical Score ling info domain 152 ER binding Class of ER bind AR binding ER binding ER, several ER binding ER binding ER binding AR binding ER reporter gene ER reporter gene ER, several ER reporter gene

17 Ref. Nr. Notes End-point a. b. c. d. e. Score 154 ER reporter gene No models 11 Same as [65] 13 No models 14 Same as [65] 17 No models 18 No models 19 No models 20 No models Same as [65] 27 No models 28 No models 29 No models No models 33 No models 34 Same as [31] 37 None - abstract only 39 No models No models 48 No models 49 No models 50 No models 14

18 Ref. Nr. Notes End-point a. b. c. d. e. Score None - no details 53 No models 54 Same as [31] 56 None - review 57 No models 58 Same as [36] 59 None - review 60 Same as [31] No models 63 No models 64 None - review 66 None - review No models 73 No models 74 No models Same as [76] 78 No models 80 None - review 81 None - review 82 None - review 83 None - review 84 None - review No models 91 None - review 92 None - review 93 Same as [36] 94 None - review 95 Same as [65] 96 No models 97 No models 99 None - review 100 Same as [36] 101 No models 102 No models 104 No models 15

19 Ref. Nr. Notes End-point a. b. c. d. e. Score No models 107 No models 109 No models 112 No models 113 No models 115 No models 116 None - review 117 No models None - review No models No models 130 No models No models No models 139 No models 141 None - abstract only 142 None - abstract only 144 No models 149 No models 155 No models This rationale scheme for applying the scores to each model permitted to identify the most promising models at the top-level scores. At the present, 9 top-scored models (in bold in Table 4) have been selected. The next task of the project (WP4) will focus on additional analysis required to identify at least one model on ER binding and one on AR binding for a further validation. Some models are available also for ER reporter gene assay but with lower scores. Among those with a score higher than 5 we will evaluate if it will be possible to validate also one of them. 16

20 WP4: Additional analyses for the sort-listed (Q)SARs For each one of the selected endpoints the preferred model (i.e. that one with the highest score) was initially chosen. Additional analyses involved the models reported in Table 5. Table 5. Selected models for the additional analyses. Selected end-point Selected model 1 2.a 2.b 3 4 ER Relative Binding Affinity (RBA) for rat Prediction of continuous values for active compounds ER classification model for RBA for rat Prediction of two classes active (any detectable activity) versus inactive AR Relative Binding Affinity (RBA) for rat CoMFA model based on continuous values ER Reporter Gene Assay Prediction of two classes active (any detectable activity) versus inactive Ref. 152 in Annex A Ghafourian & Cronin, SAR QSAR Environ. Res., 16, , 2005 Ref. 42 in Annex A Fang et al., Chem. Res. Toxicol., 14, , 2001 Ref. 118 in Annex A Tong et al., Environ. Health Perspect., 112, , 2004 Ref. 108 in Annex A Fang et al., SAR QSAR Environ. Res., 14, , 2003 New model to be published in Environ. Toxicol. Chem. by Netzeva et al. The validation procedure, when it was possible to make the complete process, was divided into two steps: a) Internal validation of the models: to accomplish the task of reproducing and verifying them and the associated statistical parameters. In this context the recalculation of chemical descriptors is very important to verify the model reproducibility and the possibility to obtain the parameters using another procedure/software (sensitivity study). b) External validation of the models: identification of suitable sources of data for the validation with new compounds; this imply an assessment of the correlation existing between new data and those used to develop the model. Another important issue is related to the definition of the applicability domain and possible solutions for defining it and the compliance of compounds used in the external set to this domain. Furthermore the transparency and reproducibility of all steps was carefully evaluated. 17

21 1: ER Relative Binding Affinity (RBA) for rat a. Activity data Ghafourian & Cronin [24] recently published a paper providing a regression model for ER rat RBA. Activity data source is the NCTRER (FDA s National Center for Toxicological Research - Estrogen Receptor Binding Database) dataset. It contains Estrogen receptor relative binding affinities for 232 chemicals tested in a radioligand competitor in vitro assay, using rat uterine cytosol, and in the paper the 131 active compounds were used (activity ranges from 4.5 to 2.6, log unit). 101 inactive compounds where with no detectable activity in the assay. Toxicity data are published in the paper and are available through several other published works [25,26] and internet sources [27]. b. Structures and chemical descriptors Chemical structures are publicly available for this dataset through the Distributed Structure-Searchable Toxicity (DSSTox) Public Database Network [28]. This SDF file (NCTRER_v2a_232_1Mar05.sdf) was used as starting point to check structures and descriptors since it is a public available source for chemical structures. Eight descriptors were used in the original equation: Name Description Type Software N C Number of carbon atoms Constitutional (1D) Tsar N halogens Number of halogen atoms Constitutional (1D) Tsar I phenol Indicator variable for phenol group (pres./abs.) Functional groups (1D) Tsar W Wiener Topological index Topological (2D) Tsar 3 a 3rd order kappa alpha shape index Topological (2D) QSARis 6 ch 6th order simple chain molecular connectivities Topological (2D) QSARis E torsion Energy of torsion for the molecule by COSMIC Force Field Quantum-chemical (3D) Energy of the highest occupied molecular orbital calculated Quantum-chemical E HOMO by VAMP (using the AM1 Hamiltonian) in TSAR (3D) Tsar Tsar c. Internal validation: model and model s performances Several linear models are reported in the paper with MLR and PLS techniques. Among them the stepwise regression model is at the same time the easiest (few descriptors) and most predictive one and was chosen for the validation. Confidence limit are also reported. Log RBA = ( ± 2. 6) ( ± 0. 05) N ( ± 0. 28) I ( ± ) 6 ( ± 2.1) ( ± 0.26) E ( ± 0. 02) ch HOMO 3 ( ± 0. 08) a ( ± 0. 08) Nhalogen C phenol E torsion W - n = 131 s = r 2 = r 2 (pred.) = F = 39.8 p = We had the original spreadsheet with the descriptors calculated in Liverpool available in our laboratory since it was produced within the framework of EASYRING project [29] which involved both laboratories. For this reason it was quite easy to reproduce the linear model published in the paper. CODESSA [30] software was used to calculate the linear regression model with the selected descriptors and the statistical parameters obtained are reported below: 18

22 Codessa Output Report : R 2 = F=39.78 s 2 = R 2 cv= X DX t-test Intercept e e N C e e N halogens e e I phenol e e W e e a e e ch e e E torsion e e E HOMO e e The MLR model coefficients and statistical performances were also confirmed with STATISTICA software [31]. In Figure 4 the plot of observed versus calculated values is reported. 3 y = 0.726x R 2 = Calculated Figure 4. Observed versus calculated values for the linear model Observed -5 Residual distribution is reported in Figure 5. For 97 compounds (74 % of the training set) residuals are within one Log unit; For 31 compounds (23.6 %) residuals are within two Log units while only for three compounds residuals are bigger. 19

23 Nr. of observations Figure 5. Residuals distribution for the training set d. External validation Up to this point the validation process permitted to verify that statistical parameters associated with the model are valid. A very critical issue for an effective and full validation of the model is related to the possibility to test the model with an external set. The first problem to be solved is to find an appropriate source of activity data for new compounds. d.1 Datasets for external validation (Pred. - Obs.) d.1.1 Inactive compounds A first check was done with 101 inactive compounds belonging to the NCTRER dataset. Of course this test set is only partially representative (from the activity point of view) of the original training set, because it involves not active compounds. On the other hand, this set was initially chosen because on an experimental point of view it is very similar to the compounds used in the training set, and also for the descriptors we had exactly those calculated in Liverpool. Six compounds were excluded since for one of them E LUMO was not calculated and for 5 compounds descriptor values resulted to be mismatched. For the remaining 95 compounds the predicted activity is reported in the histogram in Figure Nr. observations Figure 6. Residuals distribution for the test set of inactive chemicals Predicted activity 20

24 The majority of compounds (78 %) are predicted with a relatively low activity (LogRBA < -2) when the LogRBA for the active compounds ranges from 4.5 to % fall in the range from 2 to 0 and three are predicted with a very high activity. No special reasons appeared for these three wrong predictions by evaluating the applicability domain in terms of descriptor range (Table 6) or PCA score scatter plot (Figure 7). Table 6. Descriptor range and outlier descriptor values. Name Min Max Sitosterol Cholesterol 4,4'-Methylenebis (2,6-di-t-butylphenol) N C N halogens I phenol W a ch E torsion E HOMO t[2] t[1] Figure 7. PCA score scatter plot for the first two PC, calculated on the basis of the eight descriptors, for the training set (blue triangles) and the test set of inactive compounds (red triangles). Three outilers are indicated by red circles. Of course a more detailed external validation is required with another test set for evaluating false negative issue, and we will address this point in the following section. 21

25 d.1.2 Data extracted from ER1092 dataset The NTCR s center for toxicoinformatics provides through its web site access to a software for the decision forest modelling [32]. Within this software some data files are provided as examples for running the software. Two of these files are the basis of the article published in EHP. One refers to the ER232 dataset, the same under consideration here, while another one is related to a bigger dataset of 1092 compounds, ER1092: an aggregation of data from the literature for ER RBA. Anyway no further details about test conditions are provided. Unfortunately no information on CAS number or chemical compounds are provided within these files. The only identification given is mediated through the use of the AUTO_ID. An interesting finding was that this AUTO_ID is used also in the EDKB database [33], ensured by comparing molecular weights. So an external test set was prepared as followed: 1) Check of compliance of AUTO_ID for 232 substances constituting the ER232 and present also in ER ) Eliminating these compounds from the ER1092 3) Eliminating inactive compounds (AI label = 0) 4) Associating with the EDKB database the correct name and structure to the remaining compounds. This was done for compounds with an ID lower than 2248 since these are the compounds accessible via internet. As a result additional 48 compounds was found with a specified activity value and reported in Table 7. Table 7. First test set for the linear equation. AUTO_ID COMPOUND NAME CAS NUMBER MW ER combine log RBA 13 Zearalenone ,6-Didehydroisoandrosterone Nortestonate Norethindrone Norgestrel ,4'-Dihydroxybiphenyl Diamylphthalate Dicofol Naphthol ,6,7,8-Tetrahydronaphthol Tetramethylthiuram disulfide Toxaphene Benzo[a]anthracene Chlorpyrifos ,2 bis(4-hydroxy-3-methylphenyl)propane hydroxy fluorene ,2-bis(4-hydroxyphenyl)hexafluoropropane bromonaphthol monochlorobiphenyl keto-estradiol epiestriol Equilin aminophenol n-heptylphenol n-butylphenol n-pentylphenol

26 AUTO_ID COMPOUND NAME CAS NUMBER MW ER combine log RBA mercaptobenzothiazole Ziram * Captan H-Inden-1-one,2,3-diphenyl-(9CI) ,4'-(2-methylpropylidene)bisphenol Estra-1,3,5(10),6-tetraen-17-one,3-hydroxy-(8CI9CI) beta.-estradiolcyclopentylpropionate Clomiphene citrate (cis and trans mixture) Ovocyclinbenzoate beta-hydroxyestradiol Merbentul ,4'-Diphenyl-p-phenylenediamine Pharlon Di-N-hexyl phthalate Resveratrol n-hexylphenol Di-i-decyl phthalate Di-i-octyl phthalate Diheptyl phthalate Tetrabromobisphenol A Dicyclohexyl phthalate alpha-androstan-3beta-ol * Excluded compound. For this compound it was not possible to calculate one descriptor. Two of them were excluded in the analysis since one is a polymer and the other one was already present without the citrate. Finally another compound was exclude because it was not possible to calculate one descriptor. d.1.3 Japanese dataset From the literature survey it came out another possible source of data: the Japanese dataset [34]. Some limitations are implicit in the choice of these data: - data refer to human ER and not to rat; - data are obtained using a specific subtype of the receptor (ERalpha) while data source for the original model is the NCTRER dataset which used uterine cytosol ER for their experiments. In uterine cytosol alpha subtype is predominant but ER beta is also present and this can affect results especially for compounds having a selectivity for one of the subtypes such as phytoestrogens. Besides the previously reported limitations a mathematical evaluations of the correlation between the two sources of activity data (the Japanese and NCTRER) is here provided. Two series are shown in Figure 8: the first one indicated as defined contains only the 68 compounds with defined activity data in both databases. The second series contains compounds inactive at least within one of the two data sources. In the graph 5 was arbitrarily assigned as numerical value to inactive compounds. The second series contains 9 compounds which are inactive in Japanese database but active in NCTRER one (with activity values < ), 36 compounds inactive in both database and 10 compounds inactive in NCTRER database but active in Japanese one (with activity values < -1.15). Correlation line reported in the graph is calculated only on the basis of the defined series. 23

27 4 Defined n = 68 3 Japan = 948 Undefined n = y = x R 2 = Figure 8. Correlation of the activity data found in the NCTRER database, used to develop the linear model, and the Japanese database as possible source of external data for testing the equation NCTR ER = 232 Overall there is a relatively good correlation among the two database but values are relatively widespread around the ideal correlation line (within ±1 log unit). A lower consistency can be found if in the analysis inactive compounds are considered as well. Taking into account the limitations due to the degree of correlation between the two series, data with a defined activity value in the Japanese database will be used as external test set. d.2 Analysis of the descriptors used and their recalculation A very critical issue for an effective and full validation of the model is related to the possibility to recalculate the descriptors with the same software or also with other software; this will permit calculating descriptors for the external test set and evaluating if descriptor variability can affect the results of the model when applied to new compounds. For this reason the descriptors used in the equation were recalculated as explained below: 24