A METHOD FOR EVALUATING E-BUSINESS EFFECTIVENESS
A Study of eBusiness Capability in Mainland China
Jian Chen
Netec, Xi’an Software Park, Xi’an, Shaanxi, PR China
Christine Mingins
Faculty of Information Technology, Wellington Rd, Clayton, Victoria, Australia
Yan Deng
Netec, Xi’an Software park, Xi’an, Shaanxi, PR China
Keywords: e-Business evaluation, eServices, information systems effectiveness, e-Business measurement.
Abstract: Business of all kinds is increasingly being conducted on-line, potentially dissolving the boundaries between
local, national and global entities. Enterprise Information Portals are a vital tool for doing business over the
internet. No matter how sophisticated a company’s traditional business model is, its eBusiness capability is
only as effective as its online presence. It is to be expected that companies of developed companies will
perform better than companies in developing countries in this respect. This work establishes a method and
process for evaluating the effectiveness of eBusiness capabilities. The method was applied to measure the
current status of eBusiness portals of Mainland Chinese IT companies, compared with those of companies
based in developed countries. A benchmark was established by analysing the EIPs of leading international
companies and developing a method of evaluating eBusiness capability from an external, client viewpoint.
The benchmark was then used to determine the degree of difference between Chinese companies on the
basis of size, region, and business focus. This is the first systematic study on the effectiveness of e-business
applications of Mainland Chinese companies. The method and process described in this work can be used to
develop benchmarks and perform comparative evaluations of eBusiness capability for any business sector.
1 INTRODUCTION
In recent years doing business over the web has
evolved rapidly from static product advertising and
email communication to the operation of full-blown
e-Business portals (Kalakota and Robinson 2001;
Smith et al. 2003; Eikebrokk et al. 2005; UNCTAD
2004). e-Business is defined as “ ..the conduct of
business generally, with the assistance of
telecommunications and telecommunications-based
tools. e-Business therefore encompasses a wide
range of activities such as e-commerce, e-publishing
and electronic services delivery Its scope extends
across the business activities of all categories of
organisations and individuals, whether undertaken
for profit, or as a service to some community”
(Clarke 2001). Indeed, the internet economy is
rapidly evolving into a new phase where many
modular business services are provided for
customers over the web. These e-services can be
performed at a web site, on the customer’s
computer, or even computed over a grid (Casati et
al. 2001). A business will typically establish its e-
presence and engage in business with customers via
an integrated set of services accessible through a
single entry-point, an Enterprise Information Portal
(EIP) (Shilakes and Tylman 1998). These portals are
regarded as key enabling factors for small and
medium enterprises, especially in developing
countries (Eikebrokk and Olsen 2005; UNCTAD
2004; Georgiou and Steffaneas 2002).
Although there have been many reports and
research projects on applications of Internet
technology, there has been to date no specific
research on the overall level of adoption of
eBusiness applications in general, and the
effectiveness of e-Business applications in
particular, in China. The goal of the research
reported in this paper is to develop a methodology
and process for studying e-Business applications
157
Chen J., Mingins C. and Deng Y. (2006).
A METHOD FOR EVALUATING E-BUSINESS EFFECTIVENESS - A Study of eBusiness Capability in Mainland China.
In Proceedings of the International Conference on e-Business, pages 157-163
DOI: 10.5220/0001427501570163
Copyright
c
 SciTePress

used by industry sectors, and apply it to a
representative sector in Mainland China. It is
anticipated that an analysis of the differences
between best practice international businesses
operating in China and Chinese companies would
result in recommendations for improving their
business models and e-Business capabilities, in order
to improve their global competitiveness. The
perspective taken in the evaluation is that of the
customer. No matter how effective internal business
models and supporting technology is, it is the quality
of the external customer interaction with the portal
that determines the effectiveness of doing eBusiness
with the company.
2 RELATED WORK
UNCTAD (2004); Georgiou and Steffaneas (2002),
Gregorio and Kassicieh (2005), OECD (2004) and
OECD (2004a) highlight the digital divide between
developed and developing countries. An important
motivation for our research comes from the United
Nations Conference on Trade and Development, E-
Commerce and Development Report (UNCTAD
2004):
By 2003 the estimated number of internet users
globally was 276 million of whom 243 million were
from developing countries – and two thirds of those
were from Asia.
Data for e-business development in developing
countries is not easy to collect, but it is clear that
there is a distinct lag in e-Business adoption in
developing countries.
In developing countries, SMEs in urban areas
have better Internet access for operating eBusiness
applications..
Most SMEs need to have a defined eBusiness
strategy in order to adopt eBusiness applications and
hence increase their competitiveness.
Internet access is increasingly available– the
crucial issue for small business is to be able to
integrate eBusiness applications into their existing
business operations.
Best practice in eBusiness applications and the
challenges of e-Business has been discussed
extensively – for example in Moore (2001). The
current status of e-Business applications in Mainland
China is described in CNN (2004); CNNIC (2004a);
CNNIC (2004b)
Of the many studies on the evaluation of various
aspects of eBusiness, such as the adoption of and the
effectiveness of eBusiness applications (Barua et al.
2001; Eikebrokk and Olsen 2005; Yeung et al. 2003;
Vaidyanathan and Devaraj 2003; Schonberg et al.
2000; Osterwalder and Pigneur 2003). The most
relevant for our research is Pather and Remenyi
(2003).
In this work, a conceptual model for measuring
the effectiveness of eBusiness systems is proposed,
based on a combination of user satisfaction and
service quality. Our approach is consistent with the
model proposed in Pather and Remenyi (2003) in
several ways:
We evaluate the effectiveness of eBusiness
applications via EIPs on the web by evaluating the
services available in the portal and the level of the
services provided.
The level of service provision is defined based
on a benchmark of widely accepted, successful e-
Business applications.
The level of effectiveness in our approach is
defined along the same lines of user satisfaction, as
described in Pather and Remenyi. (2003) in the
sense that we regard successful eBusiness
applications as demonstrating a certain level service
quality. Gefan (2002) defines service quality as the
subjective comparison that customers make between
the quality of the services that they want to receive
and what they actually get.
3 RESEARCH METHOD AND
PROCESS
Overall we used a three-phase approach to the
project: in the first phase we determined a set of
representative international companies, operating in
China, as a benchmark to determine the commonly
used eBusiness applications, and as a basis for
developing a set of effectiveness evaluation criteria.
In the second phase a representative group of
Mainland Chinese companies was selected. In the
third phase the chosen domestic companies were
evaluated against the international benchmarks from
different perspectives.
3.1 The Research Method
The overall goal of the study was to evaluate the
effectiveness of an industry sector’s eBusiness
applications, from a user’s perspective. Enterprise
Information Portals (EIP) (in the broadest sense)
were chosen as the means to perform the evaluation,
as they provide a common application scope for
small, medium and large companies. The Chinese IT
sector was chosen as the target for the evaluation
because it is the most mature and pervasive
eBusiness sector in China.
ICE-B 2006 - INTERNATIONAL CONFERENCE ON E-BUSINESS
158

The focus of this project was to develop and apply a
method that will enable a systematic evaluation of
any industry sector. Many similar studies have used
survey and interview approaches to collect data on
the actual application and effect of these
applications; other studies discuss classical usability
techniques (inspection, enquiry, automated methods)
to evaluate the usability of web sites as an indicator
of eBusiness success (Head and Hassanein 2002).
Our approach was to establish an initial set of
attributes (eBusiness applications), relevant to the
sector under study, against which to perform the
evaluation. A set of grading criteria was established
for each application, and validated by a panel of
eBusiness experts and consultants. The following
evaluation process was applied to each EIP:
– Evaluate the EIP on its overall usability,
stability, responsiveness, and freedom from
errors.
– Compare the EIP with the set of relevant
eBusiness applications for the sector. From this,
a usage rate was calculated for each application
for the sector. This is useful not only for rating
of individual companies, but also of a whole
sector against international benchmarks.
– The effectiveness of each eBusiness application
for each selected company is then evaluated, by
actually using the application.
The scaled grading system we developed for
evaluating eBusiness effectiveness is consistent with
the notion of service quality (Pather et al. 2003). For
example, we use a 0 to 5 grading system to mark the
evaluation or the perception of using the application.
A Grade of 0 marks the absence of the eBusiness
application. A grade below 3 indicates the
expectation exceeds the perception Grade 3 indicates
that the perception matches the expectation and a
grade above 3 indicates the perception exceeds the
expectation.
Table 1 below shows an example of the grading
system. the 6 criteria and associated grades that were
applied to the application representing Applications
for Online training courses.
Table 1: Evaluation criteria for Online training courses.
Grade Criteria
5 Online application support, online
payment when applicable
4 Application form to download,
submission by fax or email, prompt
and effective reply
3 Slow responding
2 Rarely reply
1 No reply
0 No such option
3.2 The Analysis Method
Based on the evaluation data collected using the
method described above, analysis was undertaken in
three groupings:
– Evaluation of the overall interface design and
website technology of each company in the study
– Evaluation of the usage rate of e-Business
applications across different groups of
companies in the study.
– Evaluation of the effectiveness of eBusiness
applications of each company
The companies were classified according to the
following attributes: International/Domestic; Size
(small, medium, large); Location (Central,
Regional/Leading, Regional). A comparative
analysis was then performed on the different
groupings. The comparisons were made across the
following sample slices: international/domestic,
domestic by size, domestic by location, domestic by
main business focus within the IT sector.
3.3 The Process
A set of international leading software companies,
that also have major operations in Mainland China,
were selected for benchmarking against. A set of e-
Business applications applicable to all software
companies were selected. For each application an
effectiveness evaluation framework and set of
criteria were established A set of 90 domestic
software companies were selected, representing
small/medium/large, central / regional-leading /
regional, mainland businesses.
The evaluations were carried out on the
individual companies and the results analysed and
reported on according to the categories described
above. The results are therefore useful not only to
the individual companies, but also to the region,
industry sector and nationally.
4 RESEARCH FINDINGS
90 Software IT companies based in mainland China
were evaluated against a set of 40 eBusiness
applications that would be expected to be available
in a typical eBusiness portal in the software industry.
The results were benchmarked against 10
international companies with operations in China.
The complete analysis and research findings are
presented in the work of Deng et al. (2005).
A METHOD FOR EVALUATING E-BUSINESS EFFECTIVENESS -
159

4.1 Analysis Results
Table 2 below shows a representative set of results,
after grouping by size, region or type of company.
The highlighted entries represent applications with a
usage rate higher than 50% and with an
effectiveness rating of 3 or more.
Table 2: Results of the Analysis.
Application Use Rating Satis
facti
on
Large and public companies
Organization
Introduction
100% 4.3
Organization culture 100% 3.6
News 100% 4.6
Contact Information 90% 3.3
Product catalogue
and description
100% 4.4
Services and
solutions
80% 3.2
Introduction of the
services and
supports
90% 3.6
Online consultation 60% 2.6
Online technical
communication and
support
70% 3.3
Knowledge and
information
archives
50% 2.3
Downloads of
drivers, upgrade,
patches
80% 3.7
Frequently asked
questions
60% 2.2
B2C 50% 1.6
Marketing news 90% 3.8
Online recruitment 80% 3.7
Employment
policies and
organization culture
80% 2.6
Catalogue and
introduction of
partners
80% 2.1
How to be a partner 80% 3.1
Online application
for being a partner
50% 2.1
Online technical
support for partners
50% 2.4
Business and
product information
for partners
50% 2.5
13
Introduction of
training courses
70% 3.1
Medium and Regional Leading Companies
Organization
Introduction
100% 3.4 4
Organization culture 77% 2.2
News 100% 3
Contact Information 90% 2.3
Product catalogue
and description
100% 3.3
Services and
solutions
100% 3.2
Introduction of the
services and support
90% 2.6
Online consultation 57% 1.7
Online recruitment 87% 2.6
Medium and small companies
Organization
Introduction
100% 2.8
Organization culture 52% 1.2
News 92% 2.5
Contact Information 97% 1.9
Product catalogue
and description
96% 2.8
Services and
solutions
78% 2
Introduction to
services and support
64% 1.5
Online recruitment 74% 1.9
0
All domestic companies
Organization
Introduction
100% 3.1
Organization culture 66% 1.8
News 96% 2.9
Contact Information 92% 2.1
Product catalogue
and description
98% 3.1
Services and
solutions
86% 2.5
Introduction of
services and support
76% 2.1
Online recruitment 79% 2.3
2
Software companies in central cities
Organization
Introduction
100% 3.2
Organization culture 61% 2.2
News 97% 3
3
ICE-B 2006 - INTERNATIONAL CONFERENCE ON E-BUSINESS
160

Contact Information 94% 2.4
Product catalogue
and description
97% 3.2
Services and
solutions
85% 2.6
Introduction of
services and support
64% 1.9
Online recruitment 76% 2.5
Companies in the central and west region
Organization
Introduction
100% 3.1
Organization culture 65% 1.3
News 92% 2.7
Contact Information 88% 1.5
Product catalogue
and description
96% 2.8
Services and
solutions
85% 2.4
Introduction of
services and support
81% 1.9
Online recruitment 77% 2.3
Employment
policies and
organization culture
54% 1.2
Catalogue and
introduction of
partners
50% 0.9
3
General purpose Software and Services
companies in all regions of mainland China
Organization
Introduction
100 4.3
Organization culture 75% 2.5
News 50% 4
Contact Information 75% 2.8
Product catalogue
and description
100% 4.3
Services and
solutions
50% 1.8
Technical white
paper and archives
50% 1.8
Demos and online
product demos
50% 2
Introduction of the
services and
supports
75% 2.8
Online consultation 50% 1.8
Online technical
communication and
support
50% 2
Downloads of
drivers, upgrade,
patches
75% 3.3
4
Frequently asked
questions
50% 2
B2C 50% 1.3
Marketing news 50% 2.3
Online recruitment 75% 2.5
Employment
policies and
organization culture
50% 2
Catalogue and
introduction of
partners
75% 1.8
How to be a partner 50% 2
Application Software Development
companies across all regions
Organization
Introduction
99% 3.4
Organization culture 63% 1.8
News 97% 3
Contact Information 94% 2.5
Product catalogue
and description
97% 3.3
Services and
solutions
83% 2.5
Introduction of
services and support
80% 2.3
Online recruitment 80% 2.5
Catalogue and
introduction of
partners
57% 1.1
3
4.2 Analysis of Findings
Domestic companies have satisfactory performance
on interface design and website technology,
indicating they rank these factors highly, and have
the capability to achieve a satisfactory result.
There is a big difference in the effectiveness of
the applications between domestic and international
companies, especially on the level and effectiveness
of the applications.
The applications that are used widely and
relatively more effectively are those common
eBusiness applications that do not require a high
level of technical support, such as introduction of
products and services, news, product catalogues and
descriptions, service and solution introduction, basic
descriptions of customer service and support,
product white papers, software upgrade and patch
downloads and online recruitment. Although
domestic large and public companies are starting to
use applications such as online partner collaboration,
online marketing, online education and training, the
A METHOD FOR EVALUATING E-BUSINESS EFFECTIVENESS -
161

level and effectiveness of such applications are still
low.
Generally, there are some eBusiness applications
with high usage rate but with poor effectiveness. For
domestic large and public companies, such items are
mainly more advanced e-Business applications such
as customer support, B2C, Partner related
applications. This indicates that such applications
are well regarded but are not developed or deployed
effectively. Medium and small companies and
central and western region companies mainly
employ common eBusiness applications such as
product related, organization information and online
recruitment applications. The level and effectiveness
of the essential eBusiness applications of these
companies are still low. Other points worth noting
are:
– From a size perspective, large and public
companies have the highest level of eBusiness
applications.
– General purpose software companies and service
oriented companies have the highest level of
applications.
– Medium and small companies and companies in
central and western regions have the lowest level
of eBusiness applications
– Compared with other types of IT companies,
companies in the domain application
development and service area have the lowest
eBusiness capability.
Overall, we confirm that there is a large gap in
the effectiveness of overall domestic eBusiness
applications compared with best practice (leading
international companies). The difference is often at
the level of 2 grade points.
We confirm the common understanding that
large domestic companies usually perform better
than medium and small companies; however the
difference is not as great as between large mainland
Chinese domestic companies and international
companies
Companies in central cities do not necessarily
outperform companies in the eastern region
Overall, companies in the central and western
regions have the lowest eBusiness application
performance.
5 SIGNIFICANCE AND FUTURE
WORK
The IT software sector was chosen as the study
group for the application of our eBusiness
effectiveness evaluation because it would be
expected to lead in the adoption of internet based
technology. However the importance of maintaining
an effective eBusiness presence, while widely
acknowledged, is often not carried through into
practice.
e-Business technology is evolving at a rapid rate,
and it is important to be able to benchmark against
best practice in order to understand the marketplace
and to maintain a competitive advantage.
Our evaluation method enables companies to
evaluate their capabilities and effectiveness against
international and domestic best practice, and to
highlight the gaps in their online strategies. At an
industry level, this work presents an eBusiness
evaluation method and process and the foundation
for a set of eBusiness application models that can be
applied to different industry sectors.
REFERENCES
Barua, A., Konana, P., Whinston, A., and Yin, F., 2001.
Measures for E-Business Value Assessment, IT Pro,
Jan/Feb, pp47-51.
Clarke, R., 2001. If e-business is different then research in
e-business is too, IFIP Working Conference on E-
Commerce/E-Business, Salzburg, 22-23 June.
CNNIC, 2004. Report on China Internet Hotspots: Emails
and E-Commerce, 63 pages (in Chinese).
CNNIC, 2004. Report on Internet Information Resources
in China (2003).
CNNIC, 2004. Report on Regional Internet Development
Analysis of China (2004/2).
CNNIC, 2005. China Internet Development Status
Statistics and Report (2005/1), 117 pages (in Chinese).
Casati, F., Sayal, M., Shan, M-C., 2001. Developing E-
Services for Composing E-Services, Advanced
Information Systems Engineering: 13
th
International
Conference, CAiSE2001, LNCS Springer, pp.171-180.
Deng, Y., Chen, J., Mingins, C., 2005. An Empirical
evaluation of eBusiness Capability in the Chinese IT
Sector. School of Computer Science Technical report
2005/180, Monash University, 91 pages.
Eikebrokk, T.R., Olsen, D.H., 2005. Co-opetition and E-
Business Success in SME’s: an Emprical Investigation
of European SMEs, in Proceeedings of the 38
th
Hawaii
International Conference on System Sciences, pp.1-9.
Gefen, D., 2002. Customer Loyalty in e-Commerce,
Journal of the Association of Information Systems 3,
pp.27-51.
Gregorio, D.D., Kassicieh, S.K. and Neto, R.D.G., 2005.
Drivers of E-Business Activity in Developed and
Emerging Markets, IEEE Transactions on Engineering
Management, 52 (6), pp.155-166.
Georgiou, C.J. and Steffaneas, P.S., 2002. Strategies for
Accelerating the Worldwide Adoption of E-Commerce,
Communications of the ACM, 45 (4), pp.14 – 151.
Head, M. and Hassanein, K., 2002. Web Site Usability for
eBusiness Success: Dimensions, Guidelines and
ICE-B 2006 - INTERNATIONAL CONFERENCE ON E-BUSINESS
162

Evaluation Methods, Proceedings of the Hawaii
International Conference on Business, Honolulu,
Hawaii, U.S.A. June, 2002.
Kalakota, R. and Robinson, M., 2001, E-Business 2.0:
Roadmap for Success, Addison Wesley Professional.
Moore, A. (ed.) 2001, Best Practices in Enterprise
Portals, Special Supplement to KMWorld,
July/August 2001, 23 pages.
OECD (Organisation for Economic Co-Operation and
Development) 2004. Chapter 3: E-Business
Development, in OCED Information Technology
Outlook 2004, pp.105-134.
OCED (Organisation for Economic Co-Operation and
Development): ICT, E-Business and SMEs, France
2004, 46 pages.
Osterwalder, A. and Pigneur, Y., 2003. Modeling Value
Propositions in E-Business, in Proceedings of ICEC
2003, Pittsburgh, PA, USA, pp.429-436.
Pather, S., Erwin, G. and Remenyi, D., 2003. Measuring
E-Commerce Effectiveness: A Conceptual Model, in
Proceedings of the 2003 annual research conference of
the South African institute of computer scientists and
information technologists on Enablement through
Technology, pp.143 – 152.
Smith, B,R., Ackeifi, C., Bradford, T.G.,Gopalan, P.,
Maynard, J. and Mryhij, A., 2003. IBM e-business
technology, solution and design overview, IBM
Redbooks, August 2003, 198 pages.
Schonberg, E., Cofino, T., Hoch, R., Podlaseck, M. and
Spraragen, S.L., 2000. Measuring Success,
Communications of the ACM, 43 (8), pp53-57.
Shilakes, C.C and Tylman, J., 1998. Enterprise
Information Portals, Report, Merril Lynch, Inc., New
York NY, November 16.
UNCTAD 2004. e-Business and SMEs, in E-Commerce
and Development, Report, prepared by the UNCTAD
secretariat, pp.25-60.
Vaidyanathan, G. and Devaraj, S., 2003. A Five Factor
Framework for Analyzing Online Risks in E-business,
Communications of the ACM, 46 (12), pp .354-361.
Yeung, J.H.Y., Shim, J.P. and Lai, A.Y.K., 2003. Current
Progress of E-Commerce Adoption: Small and
Medium Enterprises in Hong Kong, Communications
of the ACM, 46 (9), pp.226-232.
A METHOD FOR EVALUATING E-BUSINESS EFFECTIVENESS -
163