Welcome to Evaluations Help

Return to Evaluations

Introduction: An overview of Evaluations plug-in for SFIAB

Installation: How to Install Evaluations plug-in

Process: The typical process used in Evaluations plug-in

Setup: How to set up the evaluations schemes for your event.

Step by Step: Details about the administrative actions (steps) in the Evaluations process.

Examples: Examples of setting up Evaluations for different types of project evaluations and scoring.

Tables: Details about the database tables used in Evaluations.

Apperson Scanner: How to setup SFIAB to use the Apperson Education Products Benchmark 3000 Scanner.

NOTE: The Evaluations plugin should be 'functional' at this time but not secured. Until security is incorporated, it should not be used on real data. I will need James to help implement security.
To Do list: (in order of importance - sort of):

- Anyone: Add Security. Dennis suggests the following:
  - Committe members (the administration staff) have access to all of the Evaluations package.
  - Judges can access the Evalutaion input form (page eval_webentry.php) but can enter data only if:
  They log in with their password.
  They can only enter data for teams to which they are assigned.
  (optionally perhaps every team would be required to have a team leader and only the team leader can enter data)
  This needs some planning and inputs.
- Dennis: Fix so only projects in table judges_teams_timeslots_projects_link are 'Linked' into eval_projects and eval_projects_entries. At this time all projects are linked - we will only want the 'complete' and assigned to a team projects.
- Dennis: Make sure it will work with more than one judging round. For now it is set up only for one judging round. It is unknow what will happen if there is more than one round. Perhaps an option during linking to select which round.
- Dennis: Add project number and team numbers to the exported files where usefull. add team_id and project_id to preslug export (for reference)
- Anyone: Add to TRUNK version. Dennis cannot do due to not being able to get Trunk fully running - needs help.
- Anyone: Clean up the front end user interface.
  - Set up for English or French (routines should be in sfiab_common.inc.php - if not, add them there if same as in SFIAB.
  - Make the menus and tables etc all look like SFIAB.
- Anyone: Test with different schemes - perhaps we could set up Evaluations for those current users who are interested in or currently using a scoring method.
- Anyone: Make it a true 'plug-in'. Download and add to SFIAB versus currently 'part of' SFIAB - or not
- Etc.

Introduction

Evaluations is a plug-in to SFIAB. That means you can add it to your current version of SFIAB with minimal changes to your current SFIAB version. No changes will be made to your current SFIAB data. In other words, no matter where you are in the SFIAB process for your event, you can add and take advantage of Evaluations and not worry about it creating problems. If you decide to not use it in the future you can 'unplug' it without any effect on your current setup or data.

There are two changes in the configuration of SFIAB that must be implemented during the installing of Evaluations:
SFIAB must be configured for judging as follows:
- the table config must have an entry with var = 'score_entry_enable' and val = 'yes'
- the table judges_teams_timeslots_projects_link must have a field 'score' Float length 5.2 null=yes
Configuring SFIAB in this manner will also enable the necessary links in the admin menu and the judges menu for using Evaluations. This is the SFIAB configuration used my one or more fairs in the past.

Evaluations provides a very configurable method of capturing 'scores' for your projects. You can configure multiple schemes used for scoring! For each scheme you design the evaluation criteria to use, the weighting for each criteria and the levels (choices) judges can select from for each criteria. With multiple schemes you can design a completely different method of evaluation for each different type of projects (i.e. senior divisions versus junior divisions, team projects, etc.) When your organization changes the way they want you to evaluate projects, just change the Evaluations setup.

Evaluations provides several methods of collecting your judge teams (judges) evaluations. Web pages are built for each scheme for entry on the web. CSV files are created for use with scanners or for creating your own forms to be used by the judge teams (judges)

And, all the calculations are done for you. Evaluations uses the information in your setup to calculate scores from the entries made by the judges. The code works for all combinations of possible schemes - it all driven by the tables. So, once you can set up your scheme(s) Evaluations can do the math. A unique averaging method is used when you collect more than one evaluation for each project so all the valid data is used, even if one judge team leaves out a mark or makes multiple marks on their evaluations forms (i.e. when using scanners)..

Process

The process in a nutshell is:

1. Install Evaluations plug-in:

2. Set up your evaluation schemes, criteria and values:

3. After all your projects and judge teams are configured - Link Evaluations to your SFIAB data.

4. If required, Export the necessary data for use by external programs or scanners to collect the evaluations:
- CSV files are created for download for your use in programming scanners and pre slug of the scanner forms
- CSF files can be use to create your own evaluation forms.

5. The judge teams (judges) do the evaluations and 'mark' the appropriate level for each of the criteria for the scheme that applies to the project.

- Web forms for entries are used for teams who log into the web site. Any committee member can log in, then enter the team number to bring up the projects assigned to that team.
- The team enters the evaluation for each project assigned.
- Other methods of collecting evaluations can be implemented using the exported CSV files.

6. The data is collected into a table in Evaluations.

- If the judges use a web page the data goes directly into the table.
- Uploads from a specified CSV file format will also load the table so... no matter what technique you use the data is collected in a standard format for the next steps.

7. The data is processed (a button push).

- A score is created for each judge / project assignement with complete 'captured' data.
- An average evaluation score is created for each project that is the average of all the evaluations done for the project.
- These scores can be used as desired by the fair officials

8. If a caucus of the judge team or the official responsible for a particular division requires some adjustments to the evaluations scores, that is accomplished by the team leaders (chair persons or other officials) - resulting in the final scores for the projects in each division.

9. Reports and or CSV files are available for the final results.

10. A tool is (TBD) provided to facilitate assigning the winners of some of the category awards (1st, 2nd, 3rd prizes and honorable mentions) where those winners simply rely on the divisional ranking results.

Setup

It is imperative you understand how to correctly setup Evaluations for your event. With that in mind we will try our best to explain how the setup works and the various options that are available.

Definitions:

Scheme: A method of evaluating a set of projects - perhaps differently than other projects. For instance you might have a scheme for team projects and another scheme for non-team projects. Or, you might have different schemes for each division in your event. Each scheme has a specific set of criteria, criteria weights and evaluation levels (choices) used to evaluate a project and then to calculate a score for that project. Each project is assigned a scheme. You can have as many schemes as you like. Each scheme has an id, a name and a selection statement. The selection statement is used to select which scheme is used for each project. The selection statement has three parts. the 'item', the 'condition' and the 'value'. For example 'div > 1'. Each part must be separated by one space character. Each project that matches the selection statement of one of the schemes is assigned that scheme. Here are a couple of examples of schemes:

scheme id name selection statement result
1 ISEF scoring team = no projects with one student are assigned scheme id 1
2 ISEF (Team) team = yes projects with two or more students are assigned scheme id 2

scheme id name selection statement result assumes we have 2 categories setup - Junior = 1, Senior =2
1 Junior Category cat = 1 projects with category id = 1 (junior) are assigned scheme id 1.
2 Senior Category cat > 1 projects with category id = 2 or higher are assigned scheme id 2. (could use cat = 2 also)

'item': The 'item' part can have and of the following values: (more can be added it your request but it requires a code change.)
team: If the project has one student then the value of team is 'no'. If the project has more than one student then the value of team is 'yes'
cat: This represents the setting of the category_id assigned to the project. You set up categories in SFIAB and each project gets assigned one of those categories.
div: This represents the setting of the division_id assigned to the project. You set up divisions in SFIAB and each project gets assigned one of those divisions.
grade: This represents the grade of the project's student. For teams, grade is the grade of the team member with the highest grade.
cwdiv: This represents the Canada wide science fair division id (cwsfdivisionid field in projects).

'condition': You are allowed to use any one of these symbols as the 'condition' part of the scheme selection criteria:
= Equal
!= Not Equal
< Less Than
> Greater Than

'value:' The 'value' part represents the value of the Field for each project. For instance team can have a value of 'yes' or 'no'. Cat can have values like 0,1,2... - whatever you used for the Category Ids when you set up Age Categories in SFIAB. cwsfdivisionid has values those in Canada should understand.

So, you can setup criteria like 'team = yes' or 'div = 1' or 'div < 3' or 'team != no' etc. Except for team (value = 'yes' or 'no) it is important to use a value that is set up for that field in SFIAB else the scheme_id will not be set correctly in Evaluations projects data.

Criteria: Each evaluation scheme will have a set of criteria. These are what the judge team considers when comparing a project with other projects and with a standard. Examples of criteria are: 'Scientific Thought', 'Skill', 'Thoroughness', 'Teamwork', 'Research Skill', Etc. You create the criteria for each scheme used for your event. In fact you can have different criteria and a different number of criteria for each scheme For instance, elementary school projects may require only two criteria - 'Scientific Approach' 'Completeness' where as a senior school projects will require criteria of more relevance to the project advancing to the next level.

Levels: Each evaluation scheme will have a set of levels. Levels are like grades. You determine the number of levels your evaluations will have and the value assigned to each level for each scheme. For instance you might have levels of A, B, C, D, F with values of 4,3,2,1,0 for one scheme and levels of Exceptional, Excellent, Very Good , Good, Acceptable with values of 5,4,3,2,1 for another scheme. The values do not have to be a sequence of integers but that works best. There are also optional 'special use' levels you can create that allow you to do error checking on the judges entries and take the appropriate action if the entry has errors. Special use levels are not used in calculating a score. An example is to assign a 'special use' level of 'No Entry' with a value of 0. Then when a scanner or any other process used to collect entries detects that there is no mark for a criteria it can assign the value 0 to the entry and Evaluations knows to 1.) not use that criteria entry in the calculations and 2.) indicate the error in the status for that entry. The short 4 character level_code is then added to the status for that entry. Setting a 'special use' level "MultipleMarks" with a value of 7 would allow the collection process (i.e. a scanner) to set that value when multiple entries are made for a criteria. Evaluations knows to not use that criteria's entry in the calculations and can indicate the error in the status.

Weighting: Each criteria will have a weight assigned in such a way that the sum of the weights for all the criteria to be used is = 100. Lets do an example: Say one of your evaluations schemes has 4 levels -L1, L2, L3 and L4 with values of 1,2,3,4 and the scheme has 3 criteria C1, C2 and C3. with weights of 20,30, 50. Note the weights add up to 100. Define level_value as the entry made by the judge for a given criteria. It will be 1,2,3 or 4 in this example. Also define max_level_val as the maximum level value used in calculating a score (i.e. not a 'special use' level) In this example max_level_val = 4. Given all that, a score in this example is calculated as follows:

score = ( C1.level_value * C1.weight + C2.level_value * C2.weight + C3.level_value * C3.weight) / ( max_level_val)

If all level_values are = 4 (the perfect project), then:

perfectscore = (4*20 + 4*30 + 4*50) / 4

perfectscore = 80 + 120 + 200) /4

perfectscore = 400 / 4 = 100

the lowest score for this scheme will be:

lowest score = (1*20 + 1* 30 + 1* 50) / 4 = 25

Note: You may enter a criteria with a weight of '0' This allows keeping the number of criteria the same for each scheme. This special case is used to provide compatibility with the Exported CSV files used to pre slug the Apperson Education Products Benchmark 3000 scanners.

Setup: Now that you understand (hopefully) the definitions we can explain how you set up 'Evaluations'

1.) Decide how many schemes you will need. Most events will use only one scheme, some will want two or more. Enter the schemes in the 'Create / Edit Schemes' page.

2.) For each scheme enter the evaluation criteria to be used for that scheme using the 'Create / Edit Criteria' page.

3.) For each scheme enter the evaluatons levels to be used for that scheme in the 'Create / Edit Levels' page.

When Evaluations is first installed, schemes, criteria and levels are created for the ISEF evaluations (2010) and to use the Apperson Education Products Benchmark 3000 scanners. You can use these, eliminate the 'teams' scheme or create your own schemes, criteria and levels.

Step By Step Process

Installation

To install Evaluations plug-in follow these steps:

- If the folder 'plugins' does not exist in your SFIAB website root (along with admin, data, Rmail and other folders) add the folder 'plugins'.
- Place all the folders and files frome the evaluations package under/in the folder 'plugins'.This will be the main folder 'evaluations' and under that folder many files, and some other folders (like 'db', 'uploads', 'docs'...)
- Make the evaluations folder 'uploads' writable. How to do that depends on your web server.
- When you are done you should have this setup of folders

  'root' (the base folder for sfiab)
  'admin' the SFIAB admin folder
  ... other SFIAB folders
  'plugins' ( a folder for all plugins)
  'evaluations' (the Evaluations plug-in base folder)
  'db' folder (the Evaluations data base creation and update files and the various default evaluation setup files)
  &nbs 'docs' folder (files for information or download)
  'images' folder
  'uploads' folder (uploaded files are placed here while in use - then deleted - folder must be 'writable'.)
  files... The various files for Evaluations.
  .....
  'Rmail' ... or other SFIAB folders.

- There will be no link in SFIAB to get to the main page for evaluations so.. do the following:/> - Log into SFIAB as a super user.
  - In you web browser go to 'SFIAB/plugins/evaluations/eval_index.php'
  (Where 'SFIAB' is you main browser root name for your SFIAB installation)
  - If Evaluations is not installed you will be quided through the steps to install and set up Evaluations.

What the installation does:

- Adds tables to SFIAB database. Tables eval_config, eval_projects, eval_schemes, eval_criteria, eval_levels, eval_projects_entries.
- Processes any database updates based on the Evaluations revision and sql changes.
- Proceeds to the set up where:
  - The only two changes to existing SFIAB database are made:
    - An entry is made in the SFIAB table 'config' setting var = 'score_entry_enable' with val = 'yes' (if that entry does not already exist).
    - If the SFIAB table 'judges_teams_timeslots_projects_link' does not have the field 'score' then that field is added. Score is entered as Float (5,2) (Floating point 5 digits, 2 after the decimal point. I.E. 100.00 or 44 or 66.66).
  - You You may select from one or more pre-configured evaluations schemes or enter your own evaluation scheme, criteria and levels for your event. (TBD) For now, the ISEF schemes, criteria and levels are set up.

Using Evaluations

Link Evaluations to Data base: For Evaluations to function it must be linked to the SFIAB data. This step creates (or updates) all the linkages so that Evaluations has the necessary information from SFIAB (read only) to accomplish the tasks designed into Evaluations.

Assign judge teams (or judges) to the projects they are to evaluate. The default here is to use the judge_team_projects assignments of SFIAB so it is important that these assignments are setup in SFIAB and that you run the 'Link Evaluations' after they are setup (or if they are changed) to update the Evaluations information. Optionally, you can go through a completely different assignment of judges to projects. This feature is NOT IMPLEMENTED for revision 100 of Evaluations and will be implemented only if enough requests are generated.

Create the evaluations documents or files required to enter the evaluations. This step can do several options./> &nbs 1.) A web form 'Evaluations Web Entry' created automatically for each judge teams' assigned projects using the Evaluations and SFIAB setup. The form will have the following:
- an entry for the team (or judge) number doing the evaluation. (one of the judge team members must be logged in.)
- an entry (selection) of one of the assigned project numbers.
- a web form table is presented with bubbles to mark for the project as follows
criteria criteria criteria ....
level [ ] [ ] [ ]
level [ ] [ ] [ ]
level [ ] [ ] [ ]
- a button to submit the evaluation entries

2.) CSV download files to be used by your organization to create your own web pages, or score forms or to pre slug standard forms or to import to scanners. The files should have enough data to accomplish your process. If not, request additional data be added. (coding changes)
3.) specific CSV files to be used to:
- Load the 'student' data into the Apperson Education Products Benchmark 3000 scanner (requires a specific scheme setup).
- Pre slug the Apperson Education Products Benchmark 3000 'Science Fair evaluation' forms.

Evaluate the projects: During this time the judge teams (or judges) either mark the forms for your event, enter the evaluations on the web page or some other process to capture the required information. For each project, each evaluation the following will be required:
- project number (this is automatic in web page and Apperson scanner)
- Team or judge number (this is required to verify that that team or judge has been assigned to the project.
- a list of criteria_id, level_id pairs representing the evaluation of the project. (automatic with Apperson scanner and web page)
- other optional items not yet designed.

Enter the evaluations: The data from the evaluations is entered into Evaluations tables. This is done by the web page 'Import Evaluations Data (Apperson Scanner)' or by an upload of a CSV file(s) created by you with the correct specifications. Details TBD.

Calculate the scores: This is simply a button push using page "Calculate Scores'. It calculates the individual score for each evaluation and... calculates the project's evaluation scorewhich is an average all of the evaluations for that project.

Enter the final scores: After a caucus of the judge team(s) or divisional officials and judges, some adjustment will sometimes be required to the projects scores. This can be accomplished using a Evaluations web page (TBD)

Examplesrong>

examples: TBD

Tables

TablTable descriptions TBD for now see the db.eval.full.100.sql file or browse the tables with phpmyadmin.

eval_projects: Caution: contains all projects, even those not assigned. (may change this later - or not)

eval_projects-entries. A copy of judges_teams_timeslots_projects_links with additional fields ( a record for each project - judge team assignment with the scheme_id, and an entry for the selected value_id during judging.

Setup SFIAB to use Apperson Scanner

About the Apperson Education Products Benchmark 3000 scanner:

Apperson Education Products

Apperson Education Products worked with a regional science fair in New Mexico to develop a special 'Science Fair' Option on their Benchmark 3000 scanner. This option is very robust. You download and import your fair's data into the scanners student database so that means only perfect forms will be scanned without rejection. Forms with multiple marks or missing marks are rejected but the good data is captured. Forms with a judge and exhibit combination not in the student database are also rejected. The scores collected from the forms are exported - then uploaded to your fair's Judges Evaluations for processing on the website. If you are interested in using Apperson Benchmark scanner contact Apperson Education Products. Be sure to ask for the 'Science Fair' add-on option.
Here are the requirements to use the Apperson Scanners:
- You will require a PC running Microsoft Windows and must have a license for MS Word.
- You may have no more than 99 projects in a given project Division.
- You must setup Evaluations Schemes, Criteria and Levels as described below.

Apperson Education Products - Benchmark 3000

Sample of Form A1308 - the 'science fair' score form from Apperson Education Products

Here is how the process works:

- Data from your fair is used to 'pre-slug' the score forms. A bar code with the project number and judge number (team number) is printed on the form along with other information. The bar code is not shown in the image above - it is printed to the right of the top text box marked 'SAMPLE'. A MS Word document is included in the Evaluations package as well as the file you can export to use with the MS Word document to accomplish this task using a 'Mail Merge' type process. The boxes with the word 'SAMPLE' as well as the box below 'Marking Instruction are available for your text. The supplied MS Word document and the Export file are designed to meet most needs but you can modify the document to meet your fair's needs.

- Another exported file creates the 'student data' that is loaded into the scanner. That file contains the project numbers and judge number as well as other project and judge identification information. The file is loaded into the scanner program (DataLink) and is used to verify barcodes on the forms with the data in the scanning program.

- Judges (teams) mark the forms for their assigned projects. The criteria and levels provided are those used by ISEF (as of this publish date).

- The forms are scanned and data gathered. Forms not matching the student data base are rejected and no data is created. Forms matching the student data base create a data record and are not rejected. Forms with multiple marks in any criteria or missing marks for any criteria are rejected but the data is capatured.
  - The marks on the forms create numbers corresponding to the 'value_id' as described below. 0 meaning missing mark, 7 meaning multiple marks

- The captured data records are 'exported' from the scanner at any time in the scanning process -creating a text file.

- The text files are 'imported' to SFIAB Evaluations and the data placed in a database table for processing.

- The 'Calculate Scores' selection in Evaluations processes the data and creates judge (team( scores and a project score based on the imported data.

If you want to use the Apperson Scanner to record evaluations you must setup SFIAB to meet certain conditions - as follows:

The main requirement is that the scanner (and forms) require project numbers from 1 to 9999 and Judge numbers from 1 to 9999
To accomplish this the following must be true: If the Category digit(s) are removed from the SFIAB project number the resulting numbers must be unique.
Here is one way to accomplish the above requirements:
* Set the Configuration variables in SFIAB as follows:
-Allows for the setup of different divisions for each category: = Yes (not sure, perhaps No will also work???)
-Project Numbering Format = CDN (Category, Division, Number)
* Project Age Categories: The Categories' IDs must be an integer from 1 to 9 (usually 1 = Junior, 2 = Senior)
* Divisions: The Divisions' IDs must be integers from 1 to 99 (Usually Junior = 1 to n, Senior = n+1 to ..(99max)
* When creating the Divisions, Create a separate Division ID for each Category even if the Division name is the same.
( Do not assign more than one category to a Division)
Example:
Category Division
1 - Junior 1 - Animal Science (junior)
1 - Junior 2 - Chemistry (junior)
etc ...
2 - Senior 22- Animal Science (senior)
2 - Senior 23 - Chemistry
etc
Thus, project Numbers are never more than 5 digits - i.e. CDDNN where NN is the exhibit sub number in the specific Division. And, when the 'C' is removed the remaining number is still a unique number. Project numbers used in the score form barcode and in the 'student' data are then the SFIAB project number stripped of the first digit (the category digit). However...
VERY IMPORTANT - This means there can be no more than 99 projects in a given category-division.
If that becomes a problem for your fair, a different SFIAB numbering method could be invented.
Other methods may also work. If you have a suggestion, let us know.on, let us know.
Evaluation Schemes You can have 2 - Non-Team and Team (ISEF eliminated special scoring for teams in 2011 so you may want just one evaluation scheme for all projects.)
Evaluation Criteria You must have 6 criteria for each scheme with Ids from 1 to 6 as follows *you can use ISEF weighting or your own)
id name weight non team weight team
1 -Creative Ability 30 25
2 -Scientific Thought 30 25
3 -Thoroughness 15 12
4 -Skill 15 12
5 -Clarity 10 10
6 -Teamwork 0 16
You must have 7 levels for each criteria with Ids from 0 to 7 (6 not used) as follows: (You could own 'level_values' but for ISEF the level-values are the same as the level_id.

id value name spec_use Spec_use_code
0 0 No mark yes NoMk
1 1 Acceptable no
2 2 Good no
3 3 Very Good no
4 4 Excellent no
5 5 Exceptional no
7 7 MultipleMark yes MltM
Note: level_id of 6 is not used

IMPORT data from Apperson Scanner:
The file created by the scanner has the following format (the import code is designed to use this format)
"Apperson_proj_num","Judgeteam_num","CreativeAbility.level_id","ScientificThouth.level_id","Thoroughness.level_id","Skill.level_id","Clarity.level_id","Teamwork.level_id","seq#"
wherwhere level_id are 0,1,2,3,4,5,7 0 = no mark 7 = double mark and seq# is a 6 digit sequence identifier created by the scanner.