Reviewers Management for Terra Dotta Study Abroad

All resources, including articles and video instruction, related to Reviewers Management can be found in the Reviewers Management Learning Resources article. 

Overview

Reviewers Management for Terra Dotta Study Abroad is a robust, multifaceted system which simplifies the application review process from start to finish. The use of reviewers, roles, and automation gives administrators the flexibility needed for permission assignments, notifications, and workflows.

This article discusses the following topics related to Reviewers Management:

• Reviewers Management Considerations
• Access & Permissions
• Understanding Reviewers Management
• The Reviewer Form
• Reviewer Roles
• Application in Review Trigger
• Application Submitted Trigger
• The Reviewer Console
• The Admin "Review" View
• Sample Email Notification Templates
• The Review Audit
• Frequently Asked Questions
• Webinar Recording

Reviewers Management Considerations

Note the following important information before taking advantage of the Reviewers Management system:

• It is not required that the Applicant Experience be enabled on your site in order to use Reviewers Management. However, a transition to the Application Experience is encouraged. As Terra Dotta continues to build forward with updates and new features, it will often be the case that the Applicant Experience will be required. 
• Reviewers Management must be used with the admin tools of the Admin Console. This includes the Application Finder, Application Admin Manager, Content Management, and the Reviewer Console. 
• Only reviews completed using the Reviewers Management system will appear in the Admin Console's Application Admin Manager. They will not appear in the classic admin view of an application. 
• For sites using Deployment Rules, the Reviewers Management system should be used instead of the classic reviewers functionality.
• For sites that have been using the classic Reviewers functionality, this system can be disabled by navigating to Settings > System Features > Administrative and unchecking the "Reviewers" box. This setting is only applicable to the classic reviewers system and should be disabled once you're ready to make the transition to Reviewers Management to avoid any possible confusion. 
• In addition to disabling the classic Reviewers system setting, it is recommended that reviewers in the classic "Reviewers" staff group be moved to a custom staff group with no permissions assigned. Permissions for reviewers are based on specific Reviewer Role configurations.
• When using Program Contacts, eight hard-coded program contact roles are available as options. If you would like to create a custom role, then select "Other" and manually enter the name of the custom role in the "Type Role" field that follows. This is a required step. Custom roles should be entered consistently when used in the various program contact interfaces across your site. Failure to enter the custom "Other" role type consistently will likely result in notifications not being sent as expected. 
  • Example: If you need to create a custom program contact role of "Assistant Director", then select "Other" and enter "Assistant Director" into the subsequent field. The custom role of "Assistant Director" must be entered consistently across your site when used going forward. "ASST Director", "assistant director", and "Assistant Dir." are not the same custom role as "Assistant Director".

Access & Permissions

Any Study Abroad admin user is able to access the Admin Console. However, only users with the following permissions will be able to access these features which comprise Reviewers Management:

Reviewer Roles: In order to create and add roles, these permissions are needed:

• Staff Admin: Staff Permission (edit)
• Staff Admin: Staff (add)

Reviewer Console: This feature will only appear if a user has pending or submitted reviews created with the use of Reviewers Management. The Reviewer Console is accessible via the main navigation menu, or, with the required permissions (i.e. Applicant Admin: Applicant Search (simple) and Applicant Admin: Applicant Search (advanced)), from the Applications card of the Admin Console.

A reviewer does not need to be in the "Reviewers" permission group in order to be able to access an assigned review. However, a reviewer must have a user ID in the system which is created when someone is added as a staff member or added to any staff group (including a custom group with no assigned permissions). If you have a new user whom you want to use as a program contact role as part of Reviewers Management, then you should take one of the steps listed above to ensure a user ID is created for the user. 

If you were previously using the classic Reviewers system, it is recommended that you create a custom group for your Reviewers and move them from the classic "Reviewers" group. In this way, reviewers will no longer see the classic reviewer panel on the admin home page. 

Restriction and visibility options for programs, questionnaires, and applicant parameters are not honored. Offices should use the Reviewer Role viewable elements to determine the access that reviewers have to application elements. 

Reviewer Form: This feature appears in the Content Management section of the Admin Console and is available for use by admins with the applicable Process Admin permissions:

• Process Admin: Questionnaires (View)
• Process Admin: Questionnaires (Add)
• Process Admin: Questionnaires (Edit)

Review Audit: In order to access this feature from the Applications card of the Admin Console, these permissions are needed:

• Application Admin: Reviews
• Applicant Admin: Applicant Search (simple)
• Applicant Admin: Applicant Search (advanced)

In order to perform the batch status change from the Review Audit, this permission is needed:

• Applicant Admin: Batch Application Status Changes

"Reviews" Tab in Application Admin Manager: An admin will see the addition of a "Reviews" tab when viewing an application in Application Admin Manager only after an application has been marked as Ready for Review. This tab displays a breakdown of the applicable "in progress" review workflow.

This includes a listing of the various Reviewer Roles involved along with the respective staff group, department mapping, or program contact role to which that Reviewer Role has been assigned. A counter displays the number of reviews completed of the required reviews. The option for the admin to view the first workflow as the reviewer is available by using the "Preview Review/View as Reviewer" link.

As the workflow progresses, the admin is able to see updates and gain access to viewing the review as the reviewer who is currently assigned to take action on the application. These updates include:

• The name and email address of the person who was assigned to the Reviewer Role.
• The number of submitted reviews for the role and the review decision.
• The option to view the submitted review, either the Reviewer Form or the comment provided.

Application Triggers: This feature is available for use by admins with the following permissions:

• Staff Admin: Workflow (View)
• Staff Admin: Workflow (Add)
• Staff Admin: Workflow (Edit)

Important Note: Application Triggers are available for viewing and editing by all staff with relevant permissions and not only the individual who created them as an Application Trigger rule is global across the client account.

Department Mappings: These "mappings" link applicant parameters to the "Department" field within the Department Management section of a site so that this data can be shared and used with features such as Course Approvals and the Application in Review Trigger. To use Department Mappings with the Application in Review Trigger, these site configurations are required:

• An admin user must have the staff permission of "Department Management Administrators" granted either on the group or user level in order for the "Department Management" section of the "Staff" menu to appear. 
• Under Staff > Department Management > Settings, an existing applicant parameter for both "College" and "Department" must be selected and mapped. See the Applicant Parameters Mapping section of the Course Approval Knowledgebase (KB) article for details on how to set these configurations.
• A listing of institutional colleges and departments must be created under Staff > Department Management. See the Adding Colleges and Adding Departments sections of the Course Approval KB for details on how to set these configurations. 

After these configurations have been made, you can then proceed with adding a staff member to designate as a Departmental Reviewer with these steps:

1) Navigate to Staff > Department Management > Departments.

2) Click the "V" icon at the end of the department line item for which you want to add staff. This action expands the display and prompts a "Department Members" section to appear.

3) Click the edit pencil for "Department Members".

4) Select a Member Role. From the drop-down menu for member roles, choose "Departmental Reviewer". 

5) Search for a Members. This search does not search a SIS file but instead searches for anyone who is a user in the system as a staff member. After locating and selecting the desired staff member, their name and member role will appear under Department Members.  

6) Click the check mark under the "Actions" column to apply your changes. 

7) Click the disk icon above the "Actions" column to save and preserve your changes. A confirmation message of "Departments Updated" will appear at the bottom of your page. 

Understanding Reviewers Management

The Reviewers Management system links together a cohort of features in Terra Dotta Study Abroad to deliver the flexibility admins need to build out and execute a variety of desired review workflows.

Improve your application review process in Terra Dotta with:

• A simplified reviewer permission set up with the use of Reviewer Roles and assignments.
• An automated notification system linked to Application Triggers. Reviewers know when they've been assigned an application to review, and admins are alerted when the full review workflow has been completed.
• Flexible reviewer reporting options. Assign your reviewer a customized review form to complete and/or request that they make an "Accept" or "Reject" decision. 
• A dedicated Reviewer Console that ensures your reviewers land on the correct page with the relevant information without complicated steps to access what they need. 
• Support for competitive review processes. Use the Review Audit to access a comprehensive listing of applicants and review feedback to assist in making final batch status change decisions.

To get started using Reviewers Management, familiarize yourself with this system's "cohort" of features. These include the following:

• The Reviewer Form: This is a questionnaire-like form that can be assigned to a reviewer as a means for the admin to gather more defined feedback beyond what a comment box allows. Look for the "Reviewer Form" tab in Content Management - this is where forms are created and managed. 
• The Reviewer Role: Each distinct persona involved in a review process, such as a Program Director or Academic Advisor, is known as a Reviewer Role. Rather than assigning permissions to specific people, permission-like settings are configured for each Reviewer Role.
• The Application Submitted Trigger: With the use of this trigger, it is possible for an application to be automatically marked as ready for review immediately after an application is submitted.
• The Application in Review Trigger: This trigger is used to define a set of conditions that, when met, kick off a series of review workflows (e.g. single, sequential, or both - you decide). After the full review process has been completed, a confirmation email is automatically sent to the desired recipient. 
• The Reviewer Console: From this display, a reviewer can access and manage all pending and submitted reviews that have been assigned to them. 
• The Review Audit: This feature integrates the power of Analytics to deliver reporting options from which an admin can view and access multiple application reviews at once before finalizing decisions and performing batch status change actions.

Using these features, a series of configuration steps must be taken in order to kickoff a review workflow with Reviewers Management. These are:

1) Create a Reviewer Form (as applicable).

2) Create the Reviewer Role(s).

3) Configure the Application Submitted Trigger.

4) Configure the Application in Review Trigger.

After taking action to make these required configurations and build out desired review workflows, your Reviewers Management experience is ready to begin! As soon as an application is submitted and marked as ready for review, your review process will kick off. The reviewers associated with the assigned Reviewer Roles will receive an email notification alerting them that an application is ready for review.

Note: The timing of these notifications will vary based on your review workflow configurations.

When a reviewer logs into your site, they will be directed to the Reviewer Console. This centralized display keeps all pending and submitted reviews organized for your reviewer. If a reviewer logs in using the unique "Complete Review" link sent in their notification email, then they will log in and land directly on the review page for which they need to take action.

Important Note: Reviewers with multiple profiles in the Terra Dotta system (i.e. internal and external users with the same email address) must log in using the user account to which a review has been assigned. Failure to do so will result in a mismatch between users, making the review inaccessible until the reviewer is logged in with the correct user profile.

After all reviews have been completed, a final notification is sent alerting your desired recipient that the full workflow has been completed. Then you're ready to make a final decision on your applications. With the use of desired queries and reports from Analytics, you can access a comprehensive display of a specific group of application reviews in the Review Audit. Compare reviews, make your decisions using your batch status change option - and be done!  

Activity Logging

The following actions related to Reviewers Management are recorded in an application's activity log:

• Application Review Workflow Started
• Application Reviewer Notified
• Reviewer Submitted Review
• Application Review Workflow Step Completed
• Application Review Workflow Completed
• Application Review Workflow Deleted
• Application Review Workflow Triggered But That Specific Workflow Was Already in Progress
• Application Marked Ready for Review
• Application Marked Not Ready for Review

The Reviewer Form

The Reviewer Form is a questionnaire-like form that can be created by an admin and used as a means to gather additional information about an application from a reviewer. For example, an office might take information from their review rubric and incorporate this into question items that are then answered during the review process in lieu of only receiving a general comment from the reviewer.

Clicking into the "Reviewer Forms" tab of Content Management brings you to a listing of all reviewer forms created on your site. By default, these forms will be listed from A to Z. 

Admins can conduct a search by the Reviewer Form name and use paginated search results to easily browse through a large number of items. Choose to view only "active" forms, or click the toggle to view any forms that have been retired.

Important Notes: 

• The "File Upload" question item type is not an option for use with the Reviewer Form.
• Reviewer Forms only appear in this section of Content Management and are created in a way that mirrors actions taken for building a questionnaire. The instructions for creating a Reviewer Form are therefore very similar to those for creating a questionnaire. Refer to the Questionnaire Builder KB article for additional details on question items.
• Though the Reviewer Form is a questionnaire, it does not act like a typical process element. It is not assigned to an app cycle but is rather selected as an option when building out a review workflow. Because of this, there is no cascade wizard when a change is made to a Reviewer Form. The most current version of the form is what a reviewer will see. 

Creating a New Reviewer Form

Click on the "+" sign in the lower right corner of the page to open a new Reviewer Form and build with these steps:

1) Add a title for the new Reviewer Form.

2) Add the desired instructions. A rich-text editor will populate to provide styling options. 

3) Note that the "active" toggle is enabled by default and can be used to retire the Reviewer Form.

4) Begin by creating question items. This can be done by clicking the plus "+" symbol on the bottom left of the page. An information window will appear giving you options on how to proceed. These include "Create New" and "Add Existing". 

To create a new question item, proceed in the builder using the following options:

1) Question: Write your question here.

2) Type: Use the drop-down menu to make a selection of Field, Yes or No, Date, Rating, Data Lookup, or Data Lookup with Search.

3) Sub Type: This field will appear as applicable, and a drop-down menu will provide options. For example, if you select "Field" as your type, then choose from Text, Numeric, Short Description, or Essay under Sub Type.

4) Add Instructions: These are specific to the question item.

5) Mark as Required: Click the toggle switch to mark this question item as required or not.

6) Delete: Click the trash can to remove your entry for this question item if you decide not to proceed.

Currently, a new question item will go into the "Uncategorized" folder by default and can be reused in another questionnaire.

To add an existing question item, proceed in the builder using with the following options:

1) After selecting the option to use an existing question item, a display page will appear with a full list of existing question items for you to choose from. Use the paginated search results (located on the bottom right of the page) to control the number of items which appear per page.

2) Select a question by clicking into its corresponding checkbox. More than one question item can be added at a time.

3) Once you are ready to proceed, click "Continue". If you do not want to proceed, click "Cancel".

4) After clicking continue, you will be directed back to your builder page and will see that the selected question item has been added. By default, this question item will appear last in your Reviewer Form. However, you can use the "Question Order" arrow to change the placement of this question item.

After building your first question item, you can click the plus "+" sign in the left bottom corner to create another new question item or add an existing question item to the Reviewer Form. 

5) Click "Add" to preserve your changes.

Reviewer Roles

With the use of Reviewer Roles, an admin is able to define the specific personas involved in a review workflow without having to configure a series of complex permissions.

Navigate to the main interface of Reviewer Roles to view the personas created for use with your review workflows. A "Status" column appears from which you can toggle the bar to change a role in "Active" status to that of "Deactivated".

A reviewer role is created in a three-step process as follows:

1) Navigate to Admin Console > Reviewer Roles.

2) Click on the "+" sign in the lower right corner of the page. This will open a new "Reviewer Role Configuration" display.

3) Assign your reviewer role a name in the "Role Name" field at the top of the screen. 

Example: Program Director

4) Make note that each new reviewer role is set to active status by default. To change this setting, click on the "Active" toggle switch to the right of the Role Name field.

5) Complete Step 1: Viewable Elements

A viewable element is considered to be a piece of data that a specific reviewer role has access to view.

In this step, an admin sets their configurations for what they want their reviewer to see based on their assigned role in the review workflow. For example, a reviewer role of "Program Director" might need to see different application-based elements than those assigned to a reviewer role of "Judicial Affairs Officer".

Viewable elements are divided into two categories: Standard Options and Advanced Options.

Standard Options

Some of the most common application-related data to which a reviewer is granted access appears under the Standard Options section. You can modify which viewable elements are selected by clicking the check box to the left of each respective element. 

Important Note: 

• The following standard options are automatically selected for use by default (and can be deselected if desired):
  • Applicant Name
  • Program Name
  • Program Term & Year
  • Program Location
• If the option of "Recommendations" is selected, then the reviewer will be allowed to see the recommender name, the date the recommendation was completed, the recommendation type, and the recommendation form answers and responses.
• When the option of "Attached Documents" is selected, the reviewer is only given access to view attached documents that are not marked for internal-use only.

Advanced Options

If you need to add more information for your reviewer in this role to view, click on the + icon at the bottom right of your configuration display. This will trigger a menu of Advanced Options to appear from which you with the ability to select an element. Then, with the use of a drop-down menu, specify the specific data desired.

Advanced options include the following:

• Applicant Parameters
• Assessments
• Learning Content
• Materials
• Questionnaires
• Signature Documents
• Program Parameters

Example: An admin can select the advanced option of "Applicant Parameters" and then select as many applicant parameters as desired for the specific reviewer role to have access to view. For the Program Director role, an admin might select the applicant parameters of "GPA" and "Expected Graduation Date".

Next, other Advanced Options, such as program parameters and process elements, can be assigned. 

Important Note: Assigned process elements will only display to the reviewer if they are part of the application. 

 6) Complete Step 2: Reviewer Feedback

In this step, an admin configures how they want the reviewer in this role to give feedback during the review process. The following two options are available:

Reviewer Role Form

With Reviewers Management, an admin is able to create a Reviewer Form that can be assigned to a reviewer to complete as part of a review process. This options allows the admin to collect more targeted feedback based on responses to specific question items. 

Use the drop-down menu provided to select your desired Reviewer Form.

If the use of a Reviewer Form is not required, then the admin should use the default selection of "No Form Required: Comment Box Only". With this option, a reviewer will be presented with a general comment box in which to leave remarks.

Reviewer Role Decision Action

After a reviewer leaves feedback via a comment box or Reviewer Form, they then need to make a decision to officially complete the review.

Use the drop-down menu to choose between assigning one of the following decision actions:

• Approve or Reject Review
  • The reviewer will be required to select "Approve" or "Reject" as a decision.

• Submit Review
  • The reviewer will be required to click on "Submit" as a decision.

Terminating the Review Workflow

By default, one reviewer's decision to reject a review will not terminate the entire review workflow. This is important for offices who want a sequential workflow to keep moving along, regardless of each respective reviewer's decision.

For offices with a process that requires one reviewer's decision of "reject" a review to terminate the entire review workflow, the "Rejected Review Ends Workflow" setting can be used. This setting only appears when the "Reviewer Role Decision Action" configuration is that of "Approve or Reject Workflow". Checking the box next to this setting means that if a reviewer in this role chooses to reject a review, then the entire review workflow will be terminated. An email notification (see "Review Terminated Due to Rejection" under Process > Notifications) will only be sent to the recipient configured to receive the respective Application in Review Trigger's workflow notification.

Example:

Imagine that you have created an Application in Review Trigger with a sequential workflow consisting of three reviewers:

• First, the application goes to the Judicial Affairs Officer staff group for review. 
• Next, the application goes to the Advisor program contact role for review.
• Last, the application goes to the Program Director contact role for review.

The Application in Review Trigger for this sequential workflow is configured to notify the Assistant Director of Study Abroad when the full review workflow has been completed.

The Judicial Affairs Officer (JAO) Reviewer Role has been configured so that if the JAO rejects the review, the full review workflow will be rejected.

The JAO receives an email that Nina's application is ready for review. Because Nina has had past disciplinary actions, the JAO rejects Nina's application. This "reject" decision terminates the full review workflow. Neither the Advisor nor the Program Director are notified because the communication wouldn't be relevant to either of them. Neither one of them would have known at this point that Nina had an application to review.

The Assistant Director of Study Abroad receives an automated email notification for "Review Terminated Due to Rejection" instead of the notification that the full review workflow as completed.

 7) Complete Step 3: Email Template

In this step, an email notification is configured which will be sent to this Reviewer Role when, as part of a review workflow, they have an application that has been marked as ready for review and assigned to them.

The following settings appear:

• Send Notification: This toggle will be enabled by default so that this notification is automatically sent immediately after an application has been marked as ready for review.
• Email Cc and Email Bcc: These options are deactivated by default. When enabled, a field will appear in which you can enter in the desired email address. 
  • Important Note: Email addresses must be separated by a comma with no space between email addresses. Example: [e-mail],[e-mail]
  • This field has a maximum character limit of 250.
• "From" Drop-Down Menu: Select from whom this email notification should be sent. Choose from the admin's email address, Contact 1 Email, or Contact 2 Email.
  • The email addresses for "Contact 1" and "Contact 2" are configured under Settings > Account Info.
• Notification Subject: Enter in the desired subject line.

Finally, a WYSIWYG editor appears in which you can compose your message that this Reviewer Role will see. Take advantage of using mail merge and styling options. Email templates (configured under Admin Settings) can also be applied. 

8) Click "Save & Close" to preserve your changes.

Application in Review Trigger

An admin is able to build out a review workflow and configure a notification to be received when the full review workflow has been completed with the use of the Application in Review Trigger.

See the Application Triggers KB article for general information about the Application Triggers feature of the Admin Console. 

To create an "Application in Review" trigger:

1) Navigate to Admin Console > Application Triggers. Click on the "Application in Review" tab at the top of the page.

2) Click on the "+" icon located in the bottom right corner of the interface. This will open an "Application in Review Configuration" display.

3) In the field for "Name", assign your trigger a name. 

4) From the step "1: Conditions", add the criteria for an application and its requirements that must be true in order for this trigger to apply. Click the "+" icon to view the available criteria. Options are:

• Applicant Parameter No Response: Select an applicant parameter. If there is not an assigned value for that parameter when the application is marked as ready for review, then the condition will be met.
  • You can select more than one applicant parameter, and "Or" logic will be used between each parameter. This means that if you have no response for at least one of the selected parameters, then the condition will be met. 
  • The following parameter types are not supported with this condition: Date, Number, and Type.
• Applicant Parameter Value: Select an applicant parameter, such as "Major", and the desired values for that parameter, such as "Biology" and "Mathematics". The applicant parameters and values listed are based on those configured on your site under Settings > Applicant Params. 
• Applicant Type: Select "Internal" or "External". If the trigger applies to both applicant types, then this condition should not be used as it only allows you to select one or the other, but not both.
• Application Tag: Select one or more application tags.
• Program: A "type-ahead" field allows you to find and select the name of one or more existing programs.
• Program Group: Select one or more existing program groups.
• Program Type: Select one or more program types.
• Term: A drop-down menu allows you to select one or more terms from your global program terms list (Settings > Terms).

5) From the "2: Actions" step, choose the desired action(s) which will occur when the conditions are met:

• Execute Single Workflow
• Execute Sequential Workflow
• Add Application Tag: Select one or more tags that will be applied to the application.
• Remove Application Tag: Select one or more tags that will be removed from the application.
• Set Applicant Parameter Value: Select up to one parameter and the desired value which should be automatically set for that parameter when an application meets the specified conditions of the trigger.

If you select one of the workflow options, then the next step will be to build out your desired review process. Continue reading for more details on single and sequential review workflows.

Single Workflow

When you have a review process in which you want to assign an application to a reviewer role as soon as the application has been marked as ready for review - and the order in which action is taken on the application by a reviewer is not a factor (e.g. different reviewer roles can review the application at the same time), then you would select the Single Workflow option.

 To create a single workflow:

1. Click the "+" icon from the lower right of the Application in Review configuration display. Select "Single Workflow".
2. A row for "Assign to Single Reviewer" will appear where you can set the configurations for Reviewer Role, Assigned To, and Required Reviews using a series of drop-down menus.

Reviewer Role: Make your selection based on the reviewer role required for this workflow. 

Assigned To: Use the drop-down menu to select how you want to assign your reviewer role. This can be based on one of the following options:

• Staff Group 
  • If selected, you will be prompted to choose the applicable Staff Group.
  • A staff group is created and managed from Staff >Staff Permissions.
  • This option is suggested for use when a reviewer role is used widely across programs.  

See the image above for an example.

• Program Contact Role
  • If selected, you will be prompted to choose the applicable Program Contact Role. 
  • A program contact role is created from the "Details" of a program's Program Wizard page.
  • This option is suggested for use when you have a reviewer role that is specific to a program.

• Department Mapping
  • This option should be selected when a Reviewer Role for a Department Reviewer is used. 
  • To use this option, your department mappings must first be configured. See the "Access and Permissions" section of this KB article for details.

Required Reviews: Designate how many reviews must be received for an application from this Reviewer Role. This can be based on one of the following options:

• All
  • If selected, then this means that a review must be received from everyone who has been assigned this Reviewer Role. 
  • Example A: If my Reviewer Role of Judicial Officer is assigned to a staff group of 5 people, then by selecting "All", I would need each person in the staff group to complete a review.
  • Example B: If my Reviewer Role of Judicial Officer is assigned to one program contact role, then by selecting "All", I would need one review (from this program contact role) to complete the review workflow.

• Minimum (Min.) Reviews
  • If selected, an additional column will populate from which you will need to select a numerical value. This value determines the minimum number of reviews that must be received from the Reviewer Role .
  • Example C: If my Reviewer Role of Judicial Officer is assigned to a staff group of five people, then by selecting Minimum Reviews = 3, I would need three of the five people to complete a review.
  • Example D: If my Reviewer Role of Program Director is assigned to one program contact role, then by selecting Min. Reviews = 1, this is equivalent to "Required Reviews = All". There is only one person available to give a review, so "All" is equal to the minimum value of "1" review.
  • Important Note: When a staff group is assigned a minimum number of reviews and that number is reached, then the application will no longer be available to the other members of the group who did not take action on the application. This is because their action is no longer required. If they click on the green "Complete Review" link and log in, they will be brought to a blank page. If they navigate to the Admin Console, they will not see this pending application for review in their Reviewer Console. Offices who decide to use the minimum review option might consider noting this in the email that goes out to their staff reviewers.

You can assign additional "Single Workflows" by repeating steps 1 and 2 above.

Example:

By using these two single workflows in the example above, I've built out the following review process:

• After an application is submitted and marked as ready for review, both the Department Reviewer and Judicial Officer Reviewer are notified. 
• Both Reviewer Roles can access the review from their respective Reviewer Console and take action at the same time since the review order is not a factor. 
• When the last required review is submitted, a notification email will be sent to the admin that the full review workflow has been completed.

Sequential Workflow

When you have a complex review process in which the order that different reviews occur is a factor (e.g. one review needs to be completed by a specific reviewer role before the next begins), then you would select the Sequential Workflow option. 

To create a sequential workflow:

1. Click the "+" icon from the lower right of the Application in Review configuration display. Select "Sequential Workflow".
2. A row for "Assign to Sequential Reviewer" will appear where you can set the configurations for Reviewer Role, Assigned To, and Required Reviews.
3. Click the smaller "+" icon directly to the right of this row to add a second workflow. Then repeat step two above. As you add more workflows, note that the order in which you add your workflows is the order in which they will sequentially take place.

Example:

By using the sequential workflow in the example above, I've built out the following review process:

• After an application is submitted and marked as ready for review, the Reviewer Role of "Judicial Officer" is notified. 
• After all required reviews have been completed, a notification is sent to the Reviewer Role of "Department Reviewer".
• After the required minimum review has been completed, a notification is sent to the Reviewer Role of "Program Director".
• After the required minimum review has been completed, a notification email will be sent to the admin that the full review workflow has been completed. 

Combining Workflows

It is possible to use a combination of single workflows with sequential workflows, but keep the following information in mind to ensure that your reviewers are contacted in the desired order:

• When you use a mix of sequential and simple workflows in the same Application in Review Trigger, then the first reviewer in each sequential workflow and all simple workflows will be notified at the same time to take action. 
• The "Force simple before sequential" toggle switch can be enabled to require that your single workflows be completed before any of your sequential workflows begin. This setting is deactivated by default. 
• It is not possible to force a sequential workflow to begin and finish before the simple workflows are kicked off. In this case, we would suggest using one sequential workflow. If any reviewers needed to be notified at the same time, you might put them into the same staff group and require that each person in the staff group complete a review. 

Important Note: The second step of "Actions" in an Application in Review Trigger will be locked by the system (i.e. it cannot be modified) once there is at least one review workflow in progress which uses that trigger. The workflow is locked regardless of the application's status. This means that even if a withdrawn application has an incomplete review workflow, then the related Application in Review Trigger will remain locked. For more details on locked workflows, see question #8 under the "Frequently Asked Questions" section of this article.

After completing your workflow configurations, click "Next" to continue.

6) From the "3: Notifications" step, designate to whom you want an electronic notification to be sent immediately after the full review workflow has been completed. Click the "+" icon to view the available recipient options. These are:

Program Contact: An email will be sent to one designated program contact. You must use the drop-down menu to select the desired program contact role. If you select "Other", then you must manually enter the name of this custom "Other" role. Custom roles should be entered consistently when used in the various program contact interfaces across your site. Failure to enter the custom "Other" role type consistently will likely result in notifications not being sent as expected. 

Staff Group: An email will be sent to all members of the selected staff group.

Email Address: An email will be sent to any address entered in the "Email Recipients" field. 

Important Note: See the "Notifications" section of the Admin Console: Application Triggers article for details on this feature. 

A WYSIWYG editor appears in which you can compose your message that this Reviewer Role will see. Take advantage of using mail merge and styling options. Email templates (configured under "Admin Settings") can also be applied. 

7) Click "Save & Close" to preserve your changes.

Application Submitted Trigger

With the use of the Application Submitted Trigger, an application can be automatically marked as "Ready for Review" immediately after it has been submitted. This action then kicks off the review workflow configured in your Application in Review Trigger.

To create an Application Submitted Trigger:

1) Navigate to Admin Console > Application Triggers. Click on the "Application Submitted" tab at the top of the page.

2) Click on the "+" icon located in the bottom right corner of the interface. This will open an "Application Submitted Configuration" display.

3) In the field for "Name", assign your trigger a name.

4) From the step "1: Criteria", add the criteria, or condition(s), that an application must meet when submitted in order for this rule to apply. Click the "+" icon to view the available criteria. Options are:

• Applicant Parameter No Response: Select an applicant parameter. If there is not an assigned value for that parameter when the application is submitted, then the condition will be met.
  • You can select more than one applicant parameter, and "Or" logic will be used between each parameter. This means that if you have no response for at least one of the selected parameters, then the condition will be met. 
  • The following parameter types are not supported with this condition: Date, Number, and Type.
• Applicant Parameter Value: Select an applicant parameter, such as "Major", and the desired values for that parameter, such as "Biology" and "Mathematics". The applicant parameters and values listed are based on those configured on your site under Settings > Applicant Params. 
• Applicant Type: Select "Internal" or "External". If the trigger applies to both applicant types, then this condition should not be used as it only allows you to select one or the other, but not both.
• Application Tag: Select one or more application tags.
• Phase: Select pre or post-decision.
• Program Group: A drop-down menu allows you to select one or more existing program groups.
• Program: A "type-ahead" field allows you to find and select the name of one or more existing programs.
• Term: A drop-down menu allows you to select one or more terms from your global program terms list (Settings > Terms).

Click "Next" to continue.

5) From the "2: Actions" step, choose the desired action(s) which will occur when the conditions are met and the application has been submitted:

• Mark Ready for Review: An application will automatically be marked as ready for review.
  •  When an application is marked as ready for review and it meets the assigned conditions of an Application in Review Trigger, then your configured review workflow will kick off. Admins should be aware of the connection between these two actions. 
• Add Application Tag: Select one or more application tags. 
• Remove Application Tag: Select one or more application tags.
• Set Applicant Parameter Value: Select up to one parameter and the desired value which should be automatically set for that parameter.

After the option of "Mark Ready for Review" has been selected, it will appear on your configuration display.

6) After completing step "2: Actions", it is not required that you create a notification in step "3: Notifications" when used in conjunction with an Application in Review Trigger as part of Reviewers Management. Instead, the notifications configured for each Reviewer Role will be sent to the respective reviewers at the appropriate moment during the review workflow when an application is ready for that Reviewer Role to review.

Therefore, the required next step in "3: Notifications" would be to click "Next", then "Save & Close" to preserve your changes.

Regardless, if you would like to create a notification alert for someone other than your Reviewer Roles when an application is marked as Ready for Review after being submitted, then you can do so in step "3: Notifications".  

Click the "+" icon in the lower right of the display in this step to view the available notification options. They are:

• Program Contact
• Applicant
• Email Address
• Staff Group

Important Note: See the "Notifications" section of the Admin Console: Application Triggers article for details on this feature. 

A WYSIWYG editor appears in which you can compose your message that the email recipient will see. Take advantage of using mail merge and styling options. Email templates (configured under Admin Settings) can also be applied. 

7) Click "Save & Close" to preserve your changes.

Review Workflow Only Runs Once on an Application

If an application is marked as "Ready for Review", this action kicks off the review process configured in the Application in Review Trigger. If an application was marked as "Ready for Review" by mistake, an admin can manually unmark the application as "Ready for Review". When the admin is ready to re-mark the application as "Ready for Review",  this action can be performed without duplicating the review workflow that was previously kicked off for the application.

Deleting In-Progress Workflows

If a review workflow has kicked off and a reviewer contacts the office with a need to delete their review, then the admin has the ability to perform that action from the "Reviews" tab. Clicking the "delete" icon to permanently remove a workflow triggers a warning panel to appear. This action will delete all previously submitted reviews that are part of the same workflow.

The Reviewer Console

After a reviewer has been assigned an application to review, they can quickly take action and reference their pending and submitted reviews from the Reviewer Console.

Important Note: 

• The Reviewer Console will only appear if the reviewer has a pending or submitted review.
• It is not possible for a reviewer to save their work and return at a later time to complete their review.
• After a review has been submitted, it cannot be modified in any way by the reviewer.

The Reviewer Console can be accessed in one of two ways:

• From the Configurations card of the Admin Console if the reviewer has the required permissions.
• From the main navigation menu.

The Reviewer Console contains two tabs:

• Pending Reviews: Look here for applications awaiting review.
• Submitted Reviews: Look here for applications that have been reviewed.
  • Currently, submitted reviews will remain in this tab indefinitely.

A reviewer will land on the "Pending Reviews" tab by default.

Information is organized on the Reviewer Console with columns (from left to right) as follows:

• Applicant Name
• Program Name
• App Cycle
• Decision Actions
  • A reviewer will see the option to either click "Submit" or choose between "Reject" and "Approve". What appears here to a reviewer is dependent on the option selected for the "Reviewer Feedback" step of the Reviewer Role set up.
• Open Review
  • Click the arrow to open the review in full and take action.

Click on the triangle filter icon for each column to narrow your search by keyword.

Finally, navigate through the review listing easily using the paginated search results tool.

Open Review versus Batch Action

If a Review Form has been assigned to a reviewer (as part of the Reviewer Role configuration), then the reviewer can only access this form and make a subsequent decision action by clicking the ">" icon to "Open Review". This means that the "Decision Action" column will not be populated with options and will appear as follows:

If a Review Form has not been assigned to a reviewer (as part of the Reviewer Role configuration) but instead, they are only required to leave a comment before making a decision action, then the reviewer will be presented with their assigned decision options under the "Decision Action" column as follows:

From the "Pending Reviews" tab, an admin is conveniently able to use the batch action checkbox to the left of the applicant name to select multiple applications at once on which to make a decision. 

In the example above, the admin could check the box next to the applicant's name and then select the desired decision action. A display window will appear in which the admin can leave comments before clicking on "Accept" or "Reject". Once a review has been successfully submitted in this way, the application will move from the "Pending Reviews" tab to the "Submitted Reviews" tab.

Opening a Review

After clicking on "Open Review", the full review will open in a new tab. The information displayed to a reviewer will be dependent on how the Reviewer Role was configured in step "1: Viewable Elements".

With this in mind, a reviewer may see something similar to the following:

• Assigned viewable elements appear, starting with applicant and program information. 
• A "Review" section, consisting of a reviewer form or comment box, will appear down the right side of the review display. This is followed by a decision action for the reviewer, which will be either a button to submit the review or make a determination to reject or approve the review.

Viewable elements that are process elements do not all appear in the same way to the reviewer.

Questionnaires (that have been assigned as viewable elements for the reviewer role and completed as part of the application) will appear in full, allowing the reviewer to click on the questionnaire's name to expand the field and see the applicant's responses. Files that have been uploaded via a file upload question item type will be accessible as a hyperlink.

Other process elements, such as signature documents, materials, or learning content do not display content to the reviewer. Instead, the reviewer is notified of the number of requirements that were completed as part of the application with the use of a progress counter. For example, if the applicant submitted two materials and had two materials assigned to complete, then the reviewer would see "Materials 2/2". 

After a review has been submitted, it will move from the "Pending Reviews" tab of the Reviewer Console to the "Submitted Reviews" tab.

The Admin "Review" View

After an application has been marked as ready for review, a "Reviews" tab will automatically appear within the Application Admin Manager.

From the "Review" tab, the admin can track the application's review progress. Information is organized in columns as follows:

• Reviewer Role Name
• Reviewer Role Assignment
• Review Status
  • A counter notes the number of required reviews in comparison with the number of reviews that have been submitted.

Important Note: Only reviews completed using the Reviewers Management system will appear in the Application Admin Manager. They will not appear in the classic admin view of an application. 

Sample Email Notification Templates

When setting up Reviewer Roles, you are asked to create an email notification template for each respective role in step 3. Whenever someone in a Reviewer Role has an application to review, they will be sent an email based on the notification configured in step 3 of their assigned Reviewer Role.

Here are a few sample notification templates that you might inspire you when creating your own:

In addition, when a review workflow has been completed, a notification email can be sent using the respective Application in Review Trigger.

Here is a sample notification template for the Application in Review Trigger:

As a reminder, the Review Audit is not used by reviewers. Instead, it is used by your office to see a comprehensive listing of completed reviews and perform batch status change actions.

The Review Audit

For offices using competitive review processes - or for anyone looking for an efficient way to look at all reviews for groups of applications, the Review Audit integrates with the reporting power of Analytics to provide these unique benefits:

• Access to a comprehensive listing of multiple application reviews with links that take the admin directly into the full view of an application in Application Admin Manager.
• The ability to see review scores for multiple application reviews at once, offering a more efficient way to make decisions and perform batch status change actions.
  • Important Note: The "Application ID" data point must be included in a query in order to use the batch status change action.

The Review Audit can be accessed from the Configurations card of the Admin Console if the admin has the required permissions.

Getting Started

To begin using the Review Audit, follow these steps:

1) Create your desired review query in Analytics. The most important part of this step is pulling in the data points, or "Fields", to include in the query and then selecting how you want these items grouped.

Important Note: The data point for "Application ID" must be included in your query in order to use the batch status change option.

2) Create your desired review report in Analytics. Do this using the query you created in step one. The significance of the reporting step is that you can add filters that will further define the data being displayed.

3) Navigate to the Review Audit and select your report.

The system will pull the data for this report and display it in the Review Audit for the admin to review.

From this comprehensive display, an admin can review the report data which means the ability to view multiple application reviews at once. A link is provided for direct access to an application if needed. Once you are ready to take action, use the "Batch Update Status" function.

Example 

Let's imagine that you have a competitive program called the See the World Scholarship Program

You've set up your Reviewer Role so that a Reviewer Form is used that contains numeric question items for scoring different characteristics of the application, such as the personal essay, the program fit for the applicant, and the overall content of the application.

After your full review workflow has been completed, you are notified thanks to the notification you configured using the Application in Review Trigger.

Now you want to see a comprehensive listing of all of your application reviews.

1) Create a query in Analytics.

Important Note: In addition to the Analytics KB article, a series of video instruction is available on this feature in the Analytics Learning Channel.

You might consider configuring your data points in a way that is similar to the following:

From the "Sort" column, you have the option to sort up to three data points selected. In addition, a "Calculations" feature allows you to use options such as "Sum" or "Average". For "Sum", a separate row will display at the bottom of the page with all of your numeric sums listed. 

2) Create a report in Analytics using your query.

Use the filters in the reporting tool to narrow down the data you want the system to pull for your report. In my example, I'm narrowing down by App Cycle and Program Name.

3) Navigate to the Admin Console > Review Audit. From the "Reports" menu, select your desired report. A comprehensive display of your application reviews will display. The columns displayed are based off the data points used in your query from which your report was generated.

An "Export" tab is provided to transfer the data to an Excel file if desired. 

Hyperlinked names and application IDs will appear in the listing which take the admin directly into the full view of an application in the Application Admin Manager.

4) After you have viewed your application reviews and are ready to change the status of one or more applications, select your desired applications. This will prompt the display of "Batch Action Status" button from which you can change the status of multiple applications at once. 

Frequently Asked Questions

For a list of commonly asked questions and responses related to Reviewers Management, see the Reviewers Management: Frequently Asked Questions article.

Webinar Recording

This webinar was held on August 6, 2020 and takes a closer look at the cohort of features that comprise Reviewers Management. Viewers walk through how to build out and execute a review workflow.