St

St. Gregory Thaumaturgus

Parish Center Committee

Results of the Evaluation and

Procedure for Tabulating the Data and Calculating the Results

May 26, 2009

The Parish Center Committee made two presentations which were open to the whole parish and widely publicized in the parish. The two meetings were on April 20 and 21 in the Harborcreek Fire Hall, starting at 7 PM. Registered parishioners 16 years of age and older were invited to fill in the evaluation forms. On each evening, people signed in as they entered the hall to ensure that they were registered members of the parish, and then chose a ticket with a six digit number. This number served as their ID number for tracking their personal results and provided anonymity, since they put their ID number rather than their name on the evaluation forms.

The Committee presented the final report each evening, explained their work, laid out the estimated construction costs and the pro’s and con’s of the five sites, and answered questions. In the process, the attendees each rated twelve characteristics with regard to their personal evaluation of their importance to the parish center project, rating each characteristic from 1 to 10, with 10 being most important. Then they ranked each of the five potential sites on each of the twelve characteristics, again using the 1 to 10 scale. Each meeting took in the range of 2½ to 3½ hours.

Through the parish Bulletin, the Committee also invited people to let us know if they could not attend either of these two meetings and wished to participate in the evaluation process. We wanted to ensure that everyone who wanted to participate and have a voice in this process had ample opportunity to take part. Five people did so and we walked them through the final report and filling in their evaluation forms.

There were a total of 127 evaluation forms completed: 76 from the first night’s presentation, 46 from the second night, and 5 completed later. One person turned in a form but elected NOT to evaluate, so we had 126 valid forms.

There were 12 evaluation categories to be assigned weights, and five sites to be evaluated on each of the 12 characteristics. That’s a total of (126 x 12 characteristics) plus (126 x 12 x 5 site evaluations) = 9,072 numbers.

Inputting the Data

-We made copies of the original evaluation forms and worked from the copies, to preserve the originals in case there were any questions.

-Mark Blackburn and Jim Kurre entered all data from the evaluation forms into an Excel spreadsheet.

-When numbers on the evaluation forms were not completely clear, we made a good faith effort to determine what the number was most likely to be, and used that. We did this by comparing the questionable number with other numbers on the sheet by the same person.

-If the person left a spot blank, we assigned the lowest number (1). We did this for ALL blanks.

-If the person assigned a zero, we assigned the lowest number (1).

-We built checks into the spreadsheet to alert us if we:

-missed a number somewhere.

-input a number less than 1 or greater than 10.

-did an incorrect calculation that resulted in an inappropriate number.

Tabulating the Results

1) Each person’s results were placed in a single column of the spreadsheet including:

-their ID (ticket) number

-their ranks or weights (1-10) on each of the 12 characteristics.

-their evaluations (1-10) on each of the 12 items for each of the five sites.

Checks were built in to ensure that there was a valid number (1-10) for each of these items for each person.

2) For each person, weights for the 12 characteristics were converted to percentages, totaling 100%.

This was done because each person may have assigned a mix of numbers for the weights. Some people tended to assign low numbers and some tended to assign high numbers, and we didn’t want that to cause some people’s votes to weigh more than others.

For example, consider two people who each think all 12 characteristics should be given the same weights. One person assigns all 1’s, and the other assigns all 10’s. If we just used the actual numbers as assigned, the second person’s evaluations would have 10 times the impact as the first person’s when we totaled the results across all the evaluators. That is inappropriate. Converting everyone’s weights to a 100% basis removes that potential bias and gives everyone’s evaluation the same total weight in the final calculations.

3) A weighted evaluation of each site was calculated for each person.

For the first site (Brickyard Road), the first person’s (percentage) weight on the first characteristic (Aesthetically Appealing) was multiplied by their evaluation of that characteristic at that site. That resulted in a weighted score for that site for that characteristic. This was done for each of the 12 characteristics for Brickyard Road for that person, and the result was summed. That gives their evaluation score for the Brickyard Road site. We did the same thing for each site, for each person. This process means that the weighted evaluation scores can range from one to ten, with one being the lowest (least preferred site) and ten being the highest (most preferred site.)

4) The average score for each site was calculated across all 126 evaluators, and this average score was used to rank the five sites from highest average score to lowest. Highest score is the most preferred site for the 126 evaluators.

Results

All five sites, ranked from highest to lowest:

Average Score People Ranking This Site*

Rank Site (Range: 1 – 10) Highest Lowest.

1. Brickyard Road 7.36 59 (47%) 5.9 (5%)

2. Sunset Drive 7.15 47 (37%) 8.1 (6%)

3. Main Street 5.22 17.5 (14%) 8.3 (7%)

4. Ferraro Ford 4.86 1 (1%) 25.3 (20%)

5. Pearl Street 3.78 1.5 (1%) 78.3 (62%)

Total 5.68 126 (100%) 126 (100%)

*Note: if a person’s scores resulted in a tie between two sites, each was assigned a value of 0.5 for this calculation. For example, one person had a tie between Main Street and Pearl Street, so Pearl Street had 1½ first place votes, and Main Street had 17½ first place votes. If there was a tie among 3 sites, each was assigned .33, and among 4 sites, 0.25. Three- or four-way ties sometimes happened for the lowest-ranked site.

The average score was 5.68 over all sites, all evaluators. (n = 630 = 5 sites x 126 evaluators)

The lowest score was 1.0 and the highest was 10.0. (Some people gave a site all 1’s or all 10’s on all 12 characteristics.)

Average ratio of highest site’s score to lowest site’s score: 3.58

In other words, on average, people rated their highest site about 3.5 times as good as their lowest site. This ratio ranged from 1.17 to 10.0. The smaller the ratio, the more similar a person rated the sites; the larger the ratio, the more different a person rated the sites.

The following table shows the importance people placed on the 12 characteristics. On average, they assigned 10.2% weight to “Reasonable building costs” and 10.1% to “Safe and easy access”. These were the characteristics they think most important. On the other end of the spectrum, they only assigned 2.8% to “Walking distance from home” and 5.8% to “Connects with the past.”

The two tables below show how people ranked the five sites on each of the 12 characteristics.

In the first table, #1 (green) in a cell means it ranked the highest of the five sites; #5 (red) means it ranked lowest. A high concentration of green means that this site ranked highly in several characteristics, and a lot of red means the opposite. In this table the twelve characteristics are ranked by the average weight assigned by the evaluators, so items near the top were more important to the evaluators. In the second table the characteristics are listed alphabetically and the cells present the weighted scores for each item.

The Committee has prepared an Excel file that includes all the numerical results, along with the original data from the evaluation forms. Each evaluator can find his/her own data there using the ticket that they selected on the night of the evaluation. That will let them view their own personal results in terms of the original data they chose, along with the percentage weights those numbers implied, and the site selection choices that they result in. And anyone in the parish can use the original data to double-check our calculations and results, in keeping with our policy of transparency.

If Parish Council would like us to make a presentation of these results to the parish, we are happy to do that. Once any final presentations are completed, that will conclude the work of the Parish Center Committee. We will be happy to share data and consult with the next committee who will pick up the ball at this point.

Respectfully submitted,