EVALUATION OF A COMPUTER SUPPORT-BASED CROSS

DISCIPLNE RESEARCH CONSORTIUM

Reghu Anguswamy and William B. Frakes

Research Methods Consortium, Virginia Tech., 7054 Haycock Road, Falls Church, Virginia, U.S.A.

Keywords: Graduate education, Research support, Website support, Statistical packages, Survey.

Abstract: This paper presents the results of a survey of users of services of a computer support-based cross discipline

research consortium. Usage statistics of the consortium resources is also presented. An ishikawa diagram

and analysis for improving the services is also provided. The major challenge of the consortium is

improving the visibility of its services and resources. The majority of the respondents had not heard of the

consortium, while some who did know of the consortium were not aware of what it can offer. Some

respondents were also concerned that the resources were not specific to their area of interest. However, the

majority (84%) of the respondents who had used the resources were satisfied.

1 INTRODUCTION

This paper presents a follow-up to a paper presented

at CSEDU’10 (Frakes et al., 2010). The previous

study presented a cross discipline Research Methods

Consortium for the Virginia Tech National Capital

region (NCR-RMC). Part of the NCR-RMC services

include web based, freeware, and other tools to

support the effort in an off-campus setting, with

several locations and many part-time professional

students. The purpose of the NCR-RMC is to

provide better support for research at the Virginia

Tech., National Capital Region (VT-NCR) by

coordinating research resources across colleges.

VT-NCR was established in 1969. Currently

within a 50 mile radius, there are about 18

recognized centers representing six colleges of VT-

NCR at the National Capital Region: Architecture

and Urban Studies, Business, Engineering, Science,

Liberal Arts and Human Sciences, and Natural

Resources. More than 45 graduate degree and

certificate programs are offered at VT-NCR

currently. For the current academic year, there are

about 1,100 students enrolled with about a quarter of

them being full-time; the rest being part-time. The

part-time students are mostly working professionals

and generally spend limited time on campus. VT-

NCR has currently 120 full-time faculty members.

In fiscal year 2007, VT-NCR reported $15.6 million

in sponsored research (NCR History NCR Virginia

Tech. 2010).

2 RESOURCES

NCR-RMC resources available to the faculty and

students of VT-NCR are discussed in this section. A

mindmap of the resources is also shown in Figure 1.

2.1 Website

A website was built on the WordPress blogging

platform (WordPress > Blog Tool and Publishing

platform, 2010) to support consortium activities. The

URL for the website is http://rmc.ncr.vt.edu. The

website is running on a regular desktop computer

(Dell, Intel Pentium 4, 3.2GHz processor), with a

standard freeware LAMP platform, using Ubuntu

Linux, Apache, MySQL and PHP (Frakes et al.,

2010). The website contains information about

research courses, funding sources, cross-discipline

research, and tools and methods for research.

2.2 Forum

An online forum was created using the platform -

Simple Machines Forum (Home of SMF: Free PHP

and MySQL forum software, 2010). The forum was

created to provide an environment where users could

not only share resources but also debate and ask for

243

Anguswamy R. and B. Frakes W..

EVALUATION OF A COMPUTER SUPPORT-BASED CROSS DISCIPLNE RESEARCH CONSORTIUM.

DOI: 10.5220/0003335102430248

In Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU-2011), pages 243-248

ISBN: 978-989-8425-50-8

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

Figure 1: Mindmap of the NCR-RMC resources (Frakes et al., 2010).

help on any research-related topic (Frakes et al.,

2010).

2.3 Support Person

A full-time Ph.D. student at the university with

sufficient background experience in computer

science, statistics, and research methods was

employed and given an office a desktop computer

and a laptop computer. The support person is

responsible for administration and maintainance of

the website and the forum. The support person also

acts as a consultant on statistics and statistical tools

such as R (The R Project for Statistical Computing,

2010), Microsoft Excel®, and SAS JMP (JMP

Software, 2010). Students also use the support

person for help in thesis and dissertation formatting,

as well as for the analysis of their data. The support

person is a teaching assistant for at least four

different graduate courses offered in the region. The

support person serves as a point of contact to the

RMC resources (Frakes et al., 2010).

3 SATISFACTION SURVEY

A survey was conducted to evaluate the satisfaction

of faculty and students with the resources and

services provided by the NCR-RMC. The

respondents were also asked to provide feedback on

the challenges involved in accessing the resources.

The respondents gave suggestions on improving the

services of the NCR-RMC.

The questionnaire was built using an online

survey tool – SurveyMonkey (SurveyMonkey: Free

online survey software & questionnaire tool, 2010).

The online questionnaire was pre-tested with 3 full-

time students, 2 part-time students, 2 alumni, and 1

faculty member. The questionnaire was made

accessible to the faculty and students by announcing

it through three consecutive VT-NCR weekly

newsletters. Every faculty member and student

joining the VT-NCR is by default subscribed to the

newsletter at the time of admission, with an option

to unsubscribe. The questionnaire was also sent to

the faculty and student e-mail list servers. Personal

emails were also sent by the support person to the

faculty and students.

There were a total of 58 responses. The

demographic information including contact

information and position held at VT-NCR was made

optional to answer. Hence, only 45 of the 58

respondents revealed their position held at VT-NCR.

The distribution of respondents’ position at VT-

NCR is as shown in Figure 2.

Respondents provided their purposes for

accessing the NCR-RMC resources. The

respondents could choose more than one purpose

and about 30% of them did so. The distribution is as

shown in Figure 3. At least half of them had not

used any resource from NCR-RMC. Of those who

used the resources, they were primarily for their

research work or as part of their course work. The

respondents who used the RMC resources also

ranked the resources in terms of the frequency of

using them. Usage frequency is shown in Figure 4.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

244

Figure 2: Distribution of the position held by the survey

respondents.

Figure 3: Purpose of accessing the NCR-RMC resources.

Figure 4: Ranking of the frequency in accessing the NCR-

RMC resources.

The website and the forum were the most used. The

support person was also used fairly frequently. The

thesis/dissertation support group was the least used.

Figure 5 shows the feedback on the challenges

involved in accessing NCR-RMC resources. A

fourth of the respondents said they had never heard

of NCR-RMC before they took the survey. About

half of them, who have heard of NCR-RMC were

not aware of the RMC activities and the services it

can offer. Among the respondents who had not used

any resources, 37% of them had not heard of RMC.

Half of them had heard of RMC, but were not aware

of what RMC can offer. A comment box was also

provided for feedback on this question. A student

responded that he/she had not heard of RMC until it

was introduced as part of one of his courses. A

faculty member mentioned having “not enough time

to explore.” A couple of them also mentioned that it

was hard to locate the required material within the

website and the forum.

Figure 5: Challenges in accessing the NCR-RMC

resources.

The respondents were asked to rate their satisfaction

with the NCR-RMC resources on a 5-point scale (0

being very dissatisfied and 4 being very satisfied).

Forty seven of the 58 respondents provided a

ranking. Nine of the 11 who did not provide a

ranking of satisfaction had not used any of the

resources. The distribution of satisfaction rating is

shown in Figure 6. More than half (59.6%)

responded as being “neither satisfied nor

dissatisfied”, but three-fourths of these had not used

any resource from NCR-RMC. One subject who had

responded “somewhat dissatisfied” also mentioned

that he had “heard of RMC, but was not aware of

what it can offer.” Another subject who responded

“somewhat dissatisfied” had used the RMC website,

forum and support person as part of his/her

coursework. However, the respondent was unhappy

that the resources were not specific to his/her area.

The respondent wanted more tutorials and materials,

similar to the ones on the website and forum, related

specifically to his/her field of work.

About 35% of them were “somewhat satisfied”

or “very satisfied” with the resources. The satisfied

respondents had primarily used RMC resources for

research (81%) and course work (56%). The

distribution of the resources used by these

respondents is shown in Figure 7. At least three-

fourths of them had used the website and more than

half had used the forum and the support person.

Among the respondents who had used at least one

resource frequently, 84% of them expressed

EVALUATION OF A COMPUTER SUPPORT-BASED CROSS DISCIPLNE RESEARCH CONSORTIUM

245

Figure 6: Satisfaction rating of the NCR-RMC resources.

Figure 7: Distribution of resources used by the satisfied

respondents.

“somewhat satisfied” or “very satisfied” and 12%

expressed “neither satisfied nor dissatisfied”.

An open ended question was also provided for

the respondents to provide feedback on improving

the accessibility of the resources. Of the 25

respondents who provided feedback, at least two-

thirds of them said that visibility and awareness of

the resources available should be increased, and

some suggestions were also given along these lines.

The most prominent suggested more active

advertising and improvement of publicity. Some also

suggested we hold more sessions to market the

resources. One-fourth suggested that the website

should be improved to provide news of most recent

activities and include more resources.

4 USAGE STATISTICS

4.1 Website

The NCR-RMC website was monitored using the

analytics tool from google (Google Analytics 2010).

The statistics provided are for the period September

’09-August ’10. During this period the website had

3,034 visits from 98 different countries/territories;

an average of 8.3 visits per day. Table 1 shows the

summary of statistics for the website. NCR

represents the National Capital Region composed of

Virginia, Maryland and the District of Columbia.

Table 1: Summary of usage statistics for the NCR-RMC

website.

Parameter Overall USA NCR

Visits 3034

2386

(78.6%)

1533

(21.4%)

NewVisitors 64% 58% 43%

Returning

Visitors

36% 42% 57%

Avg.pages/visit 1.9 1.89 2.06

Avg.timespent 1:30 1:36 2:00

More than three-fourths of the overall visits were

from within the USA. Almost two-thirds of the visits

within USA were from the NCR. The website has

overall returning visitors close to 36%. The majority

of them are from the USA and particularly NCR.

However, the average pages visited is comparatively

similar (about 2 pages/visit).

4.2 Forum

As of October 01, 2010, the forum had 77 members.

On average there are about 4.3 users online per day.

At least an average of 400 page views is done per

day. The forum has, on average, a new post every

four days. Figure 8 shows the number of visits on

the forum on a monthly basis. The maximum

number of visits was in Jan’10 (63,648) and the least

in Feb’10 (24,852). The forum has 23 boards of

discussion in 9 categories. The top 10 boards of

discussion are shown in Figure 9.

Figure 8: Forum visit trend over a year.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

246

Figure 9: Ishikawa diagram for challenges involved in improving NCR-RMC resources.

Figure 10: Top 10 discussion boards in forum.

4.3 Support Person

The support person has been the teaching assistant

for at least five courses (4 in computer science and 1

in Educational Research) with an average of 2

classes per semester. Statistical packages such as R,

SAS-JMP and |stat are introduced. Assignments on

these are also given and graded. Online survey tools

are also presented with demos. Introduction to

multivariate statistics is given in some course works.

The support person had helped a faculty member

from the Business School in two projects involving

statistical analysis using SAS-JMP. Nine students

used the support person for analysis of their data for

their theses/dissertations. Many of them were also

guided on formatting their work for final electronic

submission. The support person is also frequently

visited by students for help on statistical packages,

particularly SAS-JMP and R.

5 SUMMARY & CONCLUSIONS

As a follow up to the paper presented by Frakes et al

at CSEDU-10 (Frakes et al., 2010), this paper

presents an evaluation of the usage of NCR-RMC

resources. A survey study is presented on the

satisfaction of the users. More than half the

respondents were “neither satisfied nor dissatisfied”.

This was primarily because the majority of them had

not heard of the NCR-RMC. Also, many had heard

of NCR-RMC but were not aware of its services and

had not used any resource. One respondent who was

“somewhat dissatisfied” had not used any resource.

Another “somewhat dissatisfied” respondent was

unhappy the resources were not specific to his/her

research area. However, among the respondents who

have frequently used the resources, the majority

(84%) were satisfied.

The survey study examined the challenges

involved in improving the services of the NCR-

RMC. The challenges involved for NCR-RMC

resources are summarized in the ishikawa diagram

shown in Figure 10. The survey respondents also

provided feedback and suggestions to improve the

services and activities of the NCR-RMC. The

suggestions by respondents for improving the

services are shown in Figure 11.

The major concern among the respondents was

with the visibility of NCR-RMC. Among the

respondents who had not used any resources, 37% of

them had not heard of RMC. Half had heard of

RMC, but were not aware of what RMC can offer.

This might be because there was insufficient

communication through email list servers, less

advertising, and lesser involvement of the faculty.

However, the RMC was introduced to the faculty in

at least three faculty meetings. The students are

introduced to the RMC during their new student

orientation. Also, all activities of RMC are

announced through the weekly newsletters and

flyers are put up on the bulletin boards for RMC

events. Even though the support person had

introduced the RMC, and the statistical tools and

packages, as the teaching assistant in course works,

some subjects wanted technical sessions on

statistical packages to be organized more frequently

which would provide more visibility. An

introductory session on SAS-JMP and R was

EVALUATION OF A COMPUTER SUPPORT-BASED CROSS DISCIPLNE RESEARCH CONSORTIUM

247

Figure 11: Responses on how to improve RMC resources.

recently held. 3 faculty members, 5 full-time and 3

part-time students participated.

Many respondents also wanted to modify and

improve the website. One major concern raised was

that other web pages of the university do not have a

link to the NCR-RMC website. For example, one

respondent pointed out that the main library website

does not have a link to the NCR-RMC website.

However, the main website of VT-NCR (Northern

Virginia Center Virginia Tech. 2010) has a link to

the RMC website. The website also needs to be

updated constantly with more additional resources.

The number of hits for the forum is high. Almost

90% of the posts are by two administrators of the

forum. This shows that the visitors are primarily for

viewing content only and do not contribute to the

discussions. One way of improving contribution

from members is to encourage more faculty

members to get involved. Currently, the NCR-RMC

is being introduced and providing support in only

five courses. With more faculty involved, more

courses could be supported by RMC and have more

people contributing to the forum.

Another concern raised by the respondents was

that the “resources are not specific to their area.”

This is a broader issue as it is very difficult to have a

centralized service covering interests of all the

people. However, the areas of services could be

broadened if more faculty members were involved.

The support person is less extensively used than

the website or the forum. This is because the

campuses are widespread in the region and the

support person has an office only in one of the

campuses. Some respondents suggested increasing

the number of support persons, but none of those

who suggested had ever used the support person.

Even though the support person is approachable

through email or telephone, the respondents may in-

person consultation. One respondent mentioned the

support person as being inaccessible. However, the

respondent had also mentioned that he/she had not

heard of RMC before.

ACKNOWLEDGEMENTS

We would like to thank Dr. Karen DePauw, Vice

President and Dean for Graduate Education at

Virginia Tech., for her continued support to the

NCR-RMC effort.

REFERENCES

Frakes W. B., Belli G., Urbano J., and Anguswamy R.,

2010. Computer support for a cross discipline research

consortium. In CSEDU’10, 2

International

Confererence on Computer Supported Education,

Valencia, Spain.

Google Analytics 2010, viewed: 12 Oct 2010, http://

scholar.google.com

Home of SMF: Free PHP and MySQL forum software

2010, viewed: 12 Oct 2010, http://www.simple

machines.org

JMP Software 2010, viewed: 12 Oct 2010, http://

www.jmp.com

NCR History NCR Virginia Tech. 2010, viewed: 12 Oct

2010, http://ncr.vt.edu/History/

Northern Virginia Center Virginia Tech. 2010, viewed: 12

Oct 2010, http://ncr.vt.edu/History/

The R Project for Statistical Computing 2010, viewed: 12

Oct 2010, http://www.r-project.org

SurveyMonkey: Free online survey software &

questionnaire tool 2010, viewed: 12 Oct 2010, http://

www.surveymonkey.com/

WordPress > Blog Tool and Publishing platform 2010,

viewed: 12 Oct 2010, http://wordpress.org

CSEDU 2011 - 3rd International Conference on Computer Supported Education

248