Open Journal project: final report to eLib
This report covers the third and final year of the Open Journal project
from May 1997 to May 1998, and also reviews the achievements, results and
lessons from the project's full three year period from 1995. It is intended
that this report can be read alone as a record of the project's work, although
some technical details are referred to the previous two annual reports.
Part 1. Review of final year (year 3, 1997-98)
Part 2. Overall project review (1995-98)
Review of final year (year 3, 1997-98)
1 Year 3 Introduction
Part 1 of this report covers the third and final year of the Open Journal
project from May 1997 to May 1998.
Highlights of year 3
First openly-available Open Journal released for user testing
Link service software installed at one publisher's site
Moves to investigate projects with other individual publisher partners
New partner, BIDS, joins with a view to longer term collaboration
Link service modified to serve static links in PDF files
Major dissemination programme in final months
The striking feature of the project's final year were the initial moves
towards publisher-specific projects, as anticipated in the original project
proposal but effectively recasting the publishing framework within which
the project had worked, and consequently the reduced emphasis on the development
of supporting technology.
Summary of the year
Agreement was reached with the contributing publishers to make one of
the Open Journals, in cognitive science, developed in the project openly
accessible for a limited period. Previously user tests had been restricted
to selected, subject-specialist users. This large-scale test principally
demonstrated citation linking, suggesting this will be a dominating feature
of future e-journals and producing valuable findings with regard to usability,
quality, reliability, flexibility and efficiency. The first tentative moves
elsewhere among publishers towards citation linking services suggest these
are emerging issues that have not yet been confronted.
As part of an informal continuation strategy, a number of publisher
partners have begun to investigate link publishing strategies specifically
for their own resources. The most direct manifestations of these developments
to date were installing the link service suite of software on one publisher's
server, at the Company of Biologists, and the addition of another new partner,
BIDS, to the project with a particular remit beyond the project timeframe.
It has been notable how different other publisher's requirements have been
for linking applications.
This was one difficulty in forming an exit strategy, which can also
be attributed to other factors, such as the need for wider publisher collaboration,
better tools and greater user awareness, some of which may be discounted
in time. For this reason some of the development work and partnerships
established during the project are being sustained beyond it.
As a result of these publisher-led developments, work on supporting
technology was directed less at the broader innovations that had been anticipated
previously, and more towards refinements that would assist application
of the tools by publishers' Webmasters and which, informed by the project's
results, would support requirements newly identified by publishers. The
ability to serve static links in PDF files - typically the link service
applies links dynamically, or 'on-the-fly' - is an example of such a publisher-driven
Other planned technical developments had been predicated on extending
the existing Open Journals but were altered due to these publisher interventions.
The prospective publisher applications do not seek to develop the Open
Journals built with the information resources collected in the project.
As this became apparent, work on the two Open Journals in biology and computer
science concentrated on supporting the individual publisher models rather
than the broader scope envisaged.
Details and results of the project were widely disseminated during the
final months with presentations and workshops at national and international
meetings, published papers as well. A seminar was held for project partners.
Report against proposed year 3 deliverables
The following deliverables, accompanied here with brief comments, were
cited in the project's original proposal to FIGIT.
Completion and final testing of Web-based journal server for Company of
Biologists (COB), CAJUN and BCS publications, and Psycoloquy.
Link service software is installed ready for continuing use at COB, at
Nottingham University for use with CAJUN journals as applicable (likely
to be used for the journal Electronic Publishing - Origination, Dissemination
and Design), and at Southampton University for various applications
which will include BCS publications and Psycoloquy.
Completion and final testing of link database- and agent access to secondary
The single largest link database developed in the project was demonstrated
in the Open Journal of Cognitive Science. This used a bibliography agent
to create citation links between full-text journals and a large secondary
source, a database of abstracts from ISI's citation indexes. This Open
Journal provided the basis of the project's most extensive user test following
'open' release in May.
Testing and release of Open Journal Framework for use by other parties.
The 'framework' in this case refers to the set of linking tools rather
than the Open Journals. Although a version of these tools is installed
at the various sites described above, wider application is likely to require
commercialisation of the tools. There has been some negotiation, and this
is likely to continue, but a commercial version of the link service for
electronic journal applications, including components developed in the
project, is not imminent.
2 Year 3 Activities
Summary of previous years
To put the year's activities in context, here is a brief summary of developments
in technology, Open Journals and user tests described in previous annual
For a complete list of publisher partners and
people involved in the project, see Appendices
2 and 3.
Mission statement. To build a framework for publishing applications
enabling journals on the Web to be interlinked in ways which build on the
traditional qualities and identities of the journals, and which increases
the readers' ability to follow, search and access the literature for themed
study and research using the maximum available online resources.
Open Journals. The framework had begun to be used in the development
of three subject-based Open Journals:
User testing and evaluation. First versions (1.0) of the first two
Open Journals had been released for testing by subject specialists and
the results reported.
Technology: the interface between users, documents
The link service: enhancements and application in
The major link service developments for journal applications (e.g. PDF
linking for the Open Journal of Biology, and bibliography linking for the
Open Journal of Cognitive Science), based on the Ditributed Link Service
(DLS), were pioneered in the second year of the project. The third year
involved integration of these other software components with the link service,
and produced refinements in response to earlier user feedback, bug fixes,
and customisation for some of the publishers involved in the project.
Integration of software products. Two key issues identified in
previous reports were the problem of maintaining, upgrading and adding
new functions to the suite of software programs used in the project, and
the need for faster processing to deliver linked documents to users. One
way of tackling both issues was to redesign the different programs to work
more closely through tighter integration with the link service, and this
became a priority of the technical work in the final year, with implications
particularly for the PDF linking tools and citation agent, described below.
User feedback: an explicit proxy. The link service required users
to adjust the proxy setting manually in their browser. For many this
was inconvenient; for users in libraries or on corporate networks it was
often not permitted, for security reasons. As a result the link service
was customised to allow the proxy to be set explicitly as part of the URL,
so that the links on every Web page it served were automatically rewritten
to allow those linked items to be delivered via the proxy in turn. This
approach was tested in the latest release of the Open Journal of Cognitive
Debugging an OJ: looking for 'wormholes'. The native versions
of both primary journals used in the Open Journal of Cognitive Science,
that is, prior to application of the link service, had navigation 'wrappers'
which contained links to many different information resources and services.
To apply an explicit proxy, as well as adding new links such as citation
links, the link service had to rewrite all of these native links, which
would otherwise act as 'escape routes' to resources not included in the
Open Journal. One of the preoccupations in finalising the Open Journal
was to eliminate as many of the 'escape routes' as possible so that the
link service, acting as an explicit proxy, never lost track of the users'
sessions through HTML forms, CGI scripts, and the like.
Customisation for publishers. The model that Web publishers are
used to is of maintaining a core of data which is used to generate a static
HTML Web site. Consequently the link service approach can run into a psychological
barrier with publishers. Late in the third year the link service was altered
to allow it to be run as a batch process to create a new, offline and static
'linked archive' for the Company of Biologists. The company intends to
use this version of the software to add links that have been created by
its editorial assistants. The role of the link service is thus focussed
on manipulating the PDF files, whereas delivery of the documents is left
to the standard HTTP daemons and creation, management and storage of the
links themselves is left to bespoke webmaster-specified applications. This
fits well with the Web publishing model, reducing the barrier to takeup
of the link service software.
Link behaviours. Extra link display facilities were added to
the link service. These facilities (including the use of dynamic HTML for
creating link menus and lists of pending links) were never released as
part of an Open Journal because they depended too much on specific browser
features, although they were displayed at the seventh international World
Wide Web Conference in Australia. A further link variation, trails, was
implemented as part of ongoing involvement with the JISC-funded Nature
Relevant papers:  
The citation linking agent, first demonstrated supporting the Open Journal
of Cognitive Science during year 2, was developed further this year. Enhancements
to the agent technology were designed to generalise its application to
various journal citation styles, to increase processing speed, to improve
integration with the link service, and to work with the PDF tools.
Recognising different citation styles. There are two or three
broadly accepted styles of linking used in academic journals, but within
these styles there can be an almost endless number of subtle variants which
are often not just publisher-specific but journal-specific and evolve from
a journal's 'house style'. While an automated agent used to recognise citations
depends on the correctness and consistency of references, it can have built-in
rules, or heuristics, to identify specific styles. The heuristics, in the
form of sets of regular expressions, recognise particular patterns in the
document text as playing particular roles in a citation or bibliography
record. One of the aims of the third year was to make the application of
these patterns sufficiently adaptable to work across multiple journals.
A collection of heuristics applying to a particular journal might be called
a filter. The agent analyses the first citation to work out its format
and then selects an appropriate filter to apply to the subsequent citations
to extract the data. Initially based on rules for the journal Psycoloquy,
filters were added to enable the Open Journal agent to work with Behavioral
and Brain Sciences (both in the Open Journal of Cognitive Science),
and other journals included in the Open Journal of Computer Science. Although
the two journals above were the main targets, some work was done in applying
these techniques for less constrained situations, e.g. journals in the
BIDS Journals Online service and also for the Cognitive Science grey archive,
the eLib Cogprints project, with sufficiently promising results to justify
Improving processing speed. Since citation linking is performed
in 'real time', initiated by a user request for a document and performed
during download, speed is a critical factor in this intensive data process.
The speed of the agent was improved by:
Pre-processing page links. Another provision to improve speed was
modifying the agent to be run offline to provide pre-processed versions
of known pages. In this case users do not have to wait for the agent to
filter a paper if it is found in a cache of pre-processed papers.
porting the main agent code and most of the auxiliary caching code to C
migration to a database back-end from flat text files
Integration with other software tools. Conceived as an autonomous
processor, the agent nevertheless has to work other software processes
in journal linking applications. The agent can communicate as an external
process, but this is not always the most efficient implementation where
the other processes are known in advance. Partitioning the agent as a 'library'
of functions allows it to be used by other programs without running it
externally. In the project this approach was used to enable the agent to
be used with the link service and PDF software.
Relevant papers: 
To support maintenance and upgrades and improve speed of performance, integration
of the PDF-handling software with the link service became the priority
of this aspect of the work in the final year. As a result the strategy
of producing a separate PDF toolkit was replaced by building a PDF
module within the link service. The PDF module was extended over
this period to support much of the linking functionality supported by the
HTML modules in the link service, especially citation linking for which
a CGI resolver script was written.
PDF linking: enhancements and developments in year
Citation linking in PDF. By the end of the project's second year,
the PDF toolkit had enabled keyword links to be incorporated into PDF files
by extracting words and phrases from the PDF. This process used the generic
DLS technology to access linkbases and to determine links that could be
applied to a document. This functionality was included in the PDF module,
and was extended to recognise and link citations (in this context, 'citations'
refers to the bibliography or list of references typically occurring at
the end of an academic paper, not the citation occurring in the text).
To perform citation linking, a PDF article requested by a user is scanned
until a likely phrase delimiting the start of the reference section is
encountered. Typically this will be the word 'Reference' or 'Bibliography'.
Further checks (point size of the text, whether there are other words with
the same baseline, etc.) ensure this phrase is not part of the running
text of the article. When the reference section has been determined an
attempt is made to identify each individual citation. Regular expression
matching is used to determine the citation label (typically one or two
digits followed by a period, or a combination of letters and digits, e.g.
12. or AHO1997a).
Once the reference has been identified an HTML string is created, enabling
the minimal formatting of the reference string to be encoded. Some of this
formatting (e.g. font changes to italic or bold) can provide useful clues
in understanding the component parts of the reference string.
With the HTML reference string, the software can attempt to resolve
the reference. There are two ways of doing this. The first is to use the
citation agent developed in the project. The second is to use a simpler
technique. Although the PDF module will support the citation agent this
was somewhat unwieldy and the simpler citation mechanism was used in one
application, the Open Journal of Computer Science.
Citation matching: a CGI resolver script. The simpler citation
matching scheme involves the creation of a URL that points to a CGI script
residing on the project's Web server. The PDF module incorporated in the
link service parses the HTML string and generates a URL if the journal
title specified in the string is recognised. If there is a match a link
is created around the reference, but the destination of the link is not
resolved at this time. Instead, the link included in the PDF document points
to the CGI resolver script. The arguments to be passed to the resolver
script are encoded in the URL, and include information such as the journal,
title, volume, issue and page number - all of which are extracted from
the HTML string. When a link is activated the CGI script parses its arguments
and looks for the particular reference. If the document is known it can
be delivered to the user. If the specific document is not known the CGI
resolver script points to the home page of the specified journal.
On application in the Computer Science Open Journal, this simpler method
of citation matching did not work as well as originally hoped. The recognition
of citations and the link inclusion worked, but generating the destination
URLs to specific documents via the resolver caused problems. The
net result is that while citation links are displayed in a PDF file, following
those links frequently fails to produce the cited documents. Given the
small size of the target database, the number of citations that can be
resolved to specific documents will be limited. However, because the full
citation is not resolved at the time of matching, only when the link is
activated, this offers more links than are actually possible, consequently
giving the appearance of reduced link reliability. One solution
is to resolve the links fully at the time of matching, producing fewer
links, but working links, as implemented for the Open Journal of Cognitive
Science, although this demands much faster processing capabilities, especially
for PDF documents.
PDF processing: new features. Other enhancements to the PDF module
of the link service enable the inclusion of a 'watermark' facility and
themed trails in the bookmarks section of PDF files. With the watermark
a phrase such as 'This document has been enhanced by the OJF on day/month/year'
can be emblazoned across the first page of the PDF either vertically, horizontally
or diagonally. Trails enable a linkbase author to relate articles in a
particular sequence. When a trail linkbase is included in the user's Open
Journal preferences and an article contained in the trail is encountered,
two bookmarks, specifying the trail title and the next and previous articles
in the trail, are appended to the PDF bookmarks. Trails were first developed
for use in the collaboration with Nature.
Relevant papers: 
The Open Journals - year 3 updates
Cognitive Science: version 2.1, open release
As perhaps the most successful application and demonstration of the technology,
this Open Journal produced most data about the needs of further tools development
and produced new perspectives on the requirements of publishing applications.
The results of an important user test performed with
this Open Journal are presented later in the report.
Version 1.0 release (selected evaluators) June 1997 - see second year report
Version 2.0 (released internally at Southampton) January 1998
Version 2.1 (open release) May 1998
Developers: Steve Harris, Les Carr, Steve Hitchcock
Publishers: Stevan Harnad (in collaboration with Cambridge University
Press, and the American Psychological Association), ISI
Resources: Journals: Psycoloquy (APA); Behavioral and
Brain Sciences (CUP); ISI selective citation index
Major changes for version 2.0:
'Look and feel': faster, with a new layout, and more consistent of presentation
across different journals
Single rather than multiple browser windows
Major changes for version 2.1
Proxy in the url: fixing 'wormholes'
Direct linking to bibliographic data, bypassing reference lists in articles
Outcomes: User test results reported below
Still to be resolved
Reliability and accuracy of linking; impact of fuzzy matching approach;
how to test large dynamic resources
What next? Further development of this Open Journal is unlikely.
Although both publishers involved wish to continue working with the developer
group in some form, there is no plan to combine the same resources again.
Relevant papers:  
Web page design and linking. One of the features advocated for
this approach to linking is its flexibility with respect to the location
and ownership of linked materials. Early in the project attention on this
feature concerned likely publisher reaction, and how this approach can
work within current legal constraints, copyright and author's moral rights,
etc. Cross-publisher agreements where this approach might work, such as
announced by ISI, are emerging. A more immediate problem has appeared,
however. From a user perspective, the generation of links cannot be separated
from Web page design. Web design has evolved dramatically during the project,
with de facto standards emerging for gateways pointing at large
numbers of resources, and many simple documents now showing added features.
In these schemes links become not just activators but signifiers within
layouts. Feedback from the publishers and from initial user tests showed
that the presentation of the bibliographic data and the facilities for
navigating the database were as important as the link functionality, such
as the ability to travel forwards and backwards in the database. This indicates
that a better understanding is needed of how Web design and linking, although
typically separate processes, more so with the project's approach to third-party
link services, might converge, especially in the area of user interfaces
and 'gateways' to collected resources.
Fixing 'wormholes'. The move towards a proxy-based link server
and subsequently to an explicit proxy, or proxy-in-the-URL, has been explained
elsewhere. This Open Journal was among the motivators for these changes,
and in version 2.1 adopted the explicit proxy. It is vital in this arrangement,
because the user cedes all control over the link proxy, to avoid any links
in a document which do not include the proxy address, because these are
routes out of the Open Journal, referred to as 'wormholes', from which
the user is unable to return. For large resources, it is not a simple task
to ensure complete coverage - every contained link must include the proxy.
Web forms and search services, for example, have to be managed carefully.
Again, user testing reflected the magnitude of this task, reporting various
incidences of 'holes' which were then plugged. A more systematic approach
to linking via an explicit proxy needs to be developed. This is another
result that suggests a tighter integration is needed between Web journal
design and link service application.
Direct citation links. Another surprising result was the preference
for users to jump directly to linked materials wherever possible, rather
than via some intermediate stage. For example, originally all citations
in the main body of a paper in this Open Journal linked to the reference
list at the foot of the paper, and then from this list linked where possible
to the bibliographic (ISI) data. This approach can be modified to link
in-text citations directly to the ISI data, effectively cutting out the
need to look at the reference list. It is not clear that this user preference
would extend to direct linking to full text were this possible, because
user responses elsewhere, although not conclusive, suggest the need for
a 'metadata' level of information, such as bibliographic data, for initial
research. Clearly, this result might have very significant implications
for the current structure of academic publishing and the inter-relationship
between primary and secondary sources.
Identifying citation styles. The report has already described
how the citation agent has been adapted to recognise citation styles in
different journals, and it was this Open Journal that first highlighted
the requirements. In most near-term applications of this sort the largest
single resource is likely to be the secondary data, or bibliographic database.
One of the advantages of the link service approach is that in principle
this large database has to be prepared, indexed, etc., only once (and maintained!);
new primary journals can be linked into this resource without changing
the secondary data. This should not diminish the importance of preparing
the primary data for consistency, as necessary, or preparing the agent
to recognise the reference style of the new journal. After working successfully
with the journal Psycoloquy, the addition of Behavioral and Brain
Sciences to the Open Journal was less straightforward for the agent
application. Although some journal data were re-edited and the agent rules
supplemented, this part of the Open Journal still produced a higher number
of link errors. The precise reasons were not clear and there was not time
to investigate fully. It could be due to the fuzzy matching technique used
by the agent to locate bibliographic data, with implications for design
and reliability, but another possible explanation in this case may be inconsistent
journal data. With a large journal archive, preparing data could be a substantial
Biology: migration to a publisher
The first Open Journal to be produced, in year 1 of the project, this Open
Journal progressed beyond the laboratory. Real applications, if somewhat
limited initially in terms of the use of the link service, are in prospect.
Yet, given the resources available for linking, including gene sequence
databases as well as a good collection of primary journals, this Open Journal
did not realise its full scope as a demonstrator within the project. This
was partly due to some early discouraging results when the technology was
less mature, but also due to similar developments elsewhere, such as widespread
linking to the Medline abstracting and indexing service, as described by
BioMedNet . Medline linking examples typically
link a local copy of the Medline data rather than use distributed resources
as in the project. There were also differing views among the participants
about future directions for this work.
Version 1.0 release (selected evaluators) June 1997 - see second year report;
No further release. This Open Journal migrated to Company of Biologists
in December 1998
Developers: Les Carr, Owen Garrett
Publishers: Company of Biologists, BioMedNet Ltd, Academic Press
Resources: Journals: Development, Journal of Cell Biology
(both COB); Current Opinion journals series (BMN); Journal of
Molecular Biology (AP)
Changes since last release:
Demo version relocated to COB
Stronger emphasis on citation links to COB journals
Debugged link service for PDF applications
What next? Having begun the process of breaking this Open Journal
into its constituent resources, it was clear the original Open Journal
would not survive. The COB element at least will continue to be developed;
BioMedNet, although a close collaborator throughout, has not put forward
new proposals so far; neither has Academic Press, although that publisher
has expressed interest in citation linking as demonstrated in Cognitive
Relevant papers: 
Migration to Company of Biologists. At the start of the project's
third year, the Company of Biologists acquired its own in-house Web site
and Webmaster, presenting an opportunity to get feedback about the installation
and maintenance of the link service software in a publishing environment.
Responding, COB noted that the configuration set-up should be changed to
be like that of a Web server, and that the user interface should be easily
configured to adapt to the host site's look and feel. This resulted in
a new version of the software, hosted at the publisher's site. The Biology
Open Journal demonstrated in the project was too unfocussed as it stood
to be useful to the company, but a highly focussed sub-domain of the subject
area - the Naemoatode worm - has been identified which fits in with the
publisher's archive and development strategy. This subdomain
is being used as the basis of a new Open Journal which it is hoped wil
be released as an electronic-only 'special issue'.
Reduced citation linking. In a second iteration of the COB-hosted
Biology Open Journal a cut-down approach was developed for use within the
PDF parser so that citations in articles from specific publishers could
automatically link back to the publisher's own archives. In this case citations
to any COB-published journal article were linked back into its Web site.
This was worthwhile. In one of the journals, Development, some 16%
of citations within the last five years cite the journal itself.
Large biology linkbases require DLS/PDF debugging. It was discovered
that the large link databases available for biology could cause problems
for the PDF linking software. Investigations showed memory requirements
and processing time growing exponentially with the size of the linkbases.
Further exploration of the effect found some bugs in the DLS code and some
undocumented features in the Adobe PDF library. The result was that an
average-sized article in Development, containing 40 references,
could be processed in 1 second.
Computer Science: internal release
The Computer Science Open Journal was created in the last year of the project.
In contrast to the other Open Journals, all of the primary journals involved
were in PDF.
Trial version 0.1 (internal developers release only) May 1998
Developers: Steve Probets, David Evans, Les Carr
Publishers: John Wiley, British Computer Society, Oxford University
Press, MCB University Press, Chapman & Hall
Resources: Journals: Electronic Publishing - Origination, Dissemination
and Design (EPODD); Software Practice and Experience (SPE) (Wiley);
The Computer Journal (BCS/OUP); five volumes of Computer Abstracts
(MCB); Discrete Mathematics and Theoretical Computer Science, Journal
of Programming Languages, Object Oriented Systems (all C&H)
What next? Further development as part of new PDF publishing projects
at Nottingham University involving Wiley. Separately, there are plans to
work with the BCS and OUP to develop links for an e-journal hosted at Southampton
University, although this journal was not involved in the project. MCB
University Press has expressed a wish to investigate citation linking separately
for its own planned materials in HTML. C&H left the project after its
merger with Wolters Kluwer.
This Open Journal demonstrated the keyword linking facility of the link
service, but there were problems with citation linking in PDF, as described
above. Due to these continuing difficulties this Open Journal did not,
at formal completion of the project, show any additional features that
were not already evident in the Biology and Cognitive Science Open Journals
so it was not released outside the development team.
The approach to citation linking in PDF worked best for self-citing
(a journal paper citing another within the same journal), as shown by examples
based on the journals EPODD and SPE, although these journals,
which were made available to the project by Wiley, are not generally available
over the Web. Many references were linked but when links to journals other
than these two were activated the resolver script was unable to resolve
the references exactly and in such cases returned the journal home page
of the cited article. This two-stage citation resolving process, while
allowing articles in PDF or HTML to be processed faster initially, did
not prove successful in this Open Journal, for the reasons described above
in the PDF linking section.
Keyword linking was more successful, with many links being added to
articles. It was not uncommon for thousands of links to be added if an
article contained many occurrences of popular words in the linkbases. Additional
filtering techniques to reduce the number of links, and methods of distinguishing
between links of differing priority (coloured boxes, asterisks, etc.),
could be considered for future development where it is felt this
would reduce both the time taken to compile links into articles and the
machine overhead in performing the link inclusion.
Although PDF linking has been problematic, the widespread use of PDF
by journal publishers suggests that the problems are worth solving, and
it is likely this development will continue in conjunction with other PDF
publishing projects. The use of metadata containers for citation (and possibly
keyword) linking is likely to be considered in future work. Rather than
attempting to link to actual resources, links should be made to metadata
containers (e.g. RDF and related technologies). Metadata containers that
describe the functions (translations, structures and components) as well
as standard metadata such as author, title, journal, etc., should be investigated.
The project presented a broad message and disseminated widely, as shown
below. As a result it appears that few people involved in electronic publishing
or digital libraries were unaware of the project. It is more difficult
to gauge, however, perceptions and expectations of this diverse audience.
How much did these impressions contribute to the response when Open Journals
were eventually released for user testing, because motivating users to
comment was not easy. Were potential users afraid of the technical nature
of the project? Ironically, much of the later progress in the project involved
hiding many of the technical features from users, in response to earlier
Evaluation with users
Throughout the project various strategies were employed to attract and
Vitally, the first major user test was by personal visit, and it informed
much of what followed.
Types of user groups
Subject-specialist user groups were set up for two Open Journals.
One Open Journal was freely available for a limited time and was
widely announced to lists of possible users.
Communications with users
Personal visits. Most of the members of one, smaller group were
visited personally, and use of the Open Journal was observed and discussed.
Email and Web forms. Responses of the larger, more international
user groups was mediated via email, and subsequently by Web forms.
In all cases building these user groups was easy. Many volunteered willingly,
independently even, such was the level interest in the work. Over three
years well over 100 potential users registered to be informed of releases.
Subject specialists also appeared highly willing when approached. This
level of interest did not translate to real responses to the Open Journals,
however. The nature of the communication, and timing of releases - in some
cases there was a delay in users seeing Open Journals after signing up
- were no doubt contributory factors. So too were the Open Journals, and
a judgement has to made about the level of response and its implications
for the Open Journals. Despite this replies were generally positive and
constructive, as can be judged below.
A major release
During the year reported here the most significant, and the last, user
test was the open release of the Open Journal of Cognitive Science. This
was announced by email notices to various lists:
Due to the time-limited nature of this release, notices were not sent to
archival sources, Web journals such as Ariadne or D-Lib.
Behavioral and Brain Sciences and Psycoloquy associates mailing
listservs: lis-elib (UK), arl-ejournal (USA), lis-scitech (UK)
Open Journal project list (international)
Responses were received from over 20 users. Some lessons are drawn from
this test in the section reviewing the project. The user
responses and questionnaire are reproduced in full in Appendix 1.
Relevant paper: 
It was not expected that there would be major changes to the list of publisher
partners during the final year, but one new publisher did join, one left
for reasons unconnected with the project, and an attempt was made to obtain
permission to use data from a US publisher.
In. The new publisher to join was Bath Information and Data Services
(BIDS). Discussions on BIDS' involvement started in March 1997, and it
formally joined the project in March 1998 with a representative on the
project steering group. Among its activities BIDS is a journals aggregator,
and the aim is to apply some of the linking strategies developed in the
project across the range of journals in its Journals Online service. This
arrangement added no new resources to the existing Open Journals. Investigations
will continue beyond the project.
Out. Chapman & Hall left the project in March 1998 after
its acquisition by Wolters Kluwer. Although the project has a contact within
Kluwer, at that late stage no attempt was made to resurrect the arrangement
with the C&H journals as it would have involved finding a new representative
from the publisher.
Prospective. The Association of Computing Machinery in the US
was approached in April. An opportunity arose to apply a well-formed set
of links to a new electronic journal supported by the British Computer
Society and Oxford University Press, and the ACM was invited to collaborate.
Past experience from the project shows that trans-Atlantic collaboration
can be difficult to initiate at short notice. Since then other groups have
shown interest in cooperating in this work so the particular issue remains
live and will be pursued.
The list of publisher partners involved during
the project is included in Appendix 2.
At the conclusion all publishers were invited to submit proposals to
continue work from the project or explore new ideas - some had already
done so - and these initiatives are briefly described in Publishers:
the way forward as part of the overall project review below.
Open release of one Open Journal, and the freedom to demonstrate a working
model publicly, gave the opportunity for wider exposure through notices,
user testing, workshops and seminars, as well as continuing presentations
at conferences (Table 1). As a result, the last 4-5 months of the project
were particularly active for dissemination using a plan approved by the
project steering committee at the end of 1997.
Dissemination was targetted at a broad range of interests - developers,
researchers, scientists, teachers, publishers and librarians - in the UK
and overseas, with presentations this year given in the USA, France and
Table 1. Presentations at conferences and
other meetings 97-98
|Type of meeting
||Speakers from project
||Digital Libraries '97: ACM
||Philadelphia, PA, USA
||Open Journal publishers' seminar
||Project partners, publishers
||David Brailsford, Robert Kimberley,
||Web Database Publishing:
BCS specialist group on Electronic and Multimedia Publishing
||Technical developers, commercial
||Les Carr, Andrew Witbrock (BioMedNet)
||IRISS'98, Internet Research and
Information for Social Scientists: ILRT, Bristol University
||Social science researchers, teachers
||c. 150 (in one session)
||Robert Kimberley, Steve Hitchcock
||Annual conference: UK Serials
||Publishers, librarians, academics
||c. 30 (for two workshops)
||Steve Hitchcock, Les Carr
||Electronic Publishing '98:
||St Malo, France
||Technical: developers, researchers
||Seventh WWW Conference: IW3C2
||Technical: Web developers, researchers
||Making the most of e-journals:
||Librarians: academic and industrial
||Hypertext '98: ACM
For details of subsequent publications, see in this report
Electronic journal publishing in context. An additional strand to
dissemination has been the project's presentations and publications on
more general developments in e-journals publishing. These works were not
explicitly about the project, but were based on research intended to inform
the project of developments in the publishing environment it was targetting.
Yet publishing the results of these surveys has brought international attention
to the project. By fortune of timing the project's first e-journal survey
in 1996 received most response. A partial follow-up published during the
project's final year  was itself the result of
an invitation to extend the original survey.
Clarifying the vision: impact on exit strategy. Vision is
an important motivating factor for innovative work. It is important that
all members of a development group understand and share the same vision.
This may often appear to be the case, but can be deceptive. A primary objective
of the project was to create an Open Journal 'framework', the original
proposal describing this as a "framework of information retrieval technologies
and electronic publishing practises to be used by information providers".
That framework was to be used, among other objectives, to "develop a set
of ‘open journals’ based on existing publications". Yet at stages in the
project it may variously have been believed that the product would be a
collection of software tools, individual software tools, a service which
used these tools, or a set of link databases for distinct journals or for
journals collectively - the Open Journals. This apparent lack of clarity
particularly affected the discussion of the project's exit strategy, and
consequently the way the final year developed. One reason for the different
perceptions is that the project grew substantially - doubling the number
of publishers and contributed resources - after launch and some time after
the original proposal. New partners have an influence. The path to innovation
is never smooth and throws up new perspectives. What has been shown in
this case is that the original vision needs to be robust and strongly held
by a nucleus of participants if it is to withstand such challenges, and
the earlier it is tested the better.
3 Learning from Experience
Tackling diverse agendas: impact of exit strategy. Superficially
a project involving two types of partner, developer groups and publishers,
would not appear to be especially diverse. But publishers have been described
as notoriously anti-Procrustean; indeed, it is seen as a strength in a
competitive industry. So the project is not the first such collaboration
to have encountered the problem of competing interests and hidden agendas.
Recognising and tackling the problem, however, are different matters. Open
communication has to be encouraged, but creating the environment in which
this can happen is difficult. One way of achieving this is to place the
project interests beyond the scope of competing, or commercial, interests,
and a research project may have the chance to do this. For development
work, however, such a strategy cannot survive through a project. There
is also the demand within the eLib programme to create an exit strategy,
an understandable and important objective. It has been noted in Tavistock
reviews of the eLib programme that some projects found it difficult to
produce business plans and continue development. There is another aspect:
not just the pressure of time, but the imposition of a schedule for exit
strategies. As for the Open Journal project, there is no doubt that the
need to discuss an exit strategy to a programme-led rather than a project-led
schedule altered the scope, momentum and complexion of the final year work.
Given the structure of the project and its partners, this was a process
it obviously wanted to tackle, but it may have been more productive, and
might still be, had a business strategy been allowed to emerge more naturally,
after more testing and further releases, because this remains an area where
development and technology remain in advance of cultural and commercial
Evaluation is a good thing! Independent evaluation of a project is
vital. If there is one lesson that might be applicable to other collaborative
eLib projects, it is to appoint an external evaluator early. The evaluator
is positioned to identify fault lines - tensions and anxieties - between
collaborating groups. These lines invariably materialise over time, are
often difficult to discern by the direct participants or are concealed,
and become more difficult to tackle both because they can become embedded
in practices and procedures and because issues of ownership, of ideas as
much as products, become more acute later in a project. Such tensions may
concern major or minor issues, but will always have an effect on the progress
and efficiency with which the work is prosecuted.
In the case of the Open Journal project, the
evaluator's report is included in part 2, the overall project review.
4 Year 3 Project
Table 2. Summary of project finances from
May 1997 to the end of the project
|Subject-expert consultancy fees
1. Underspend on consultancies and salaries is committed
on a sub-contract basis for authoring, development and evaluation, either
not yet invoiced or being processed.
The project steering group, which directed the project, met five times
during the final year, 15 times overall. The steering group included representatives
from all major partners. During the year some partners brought more members
into the group as the role of meetings took on a stronger reporting function.
Attendance was invariably good, between 10 to 15 people, typically more
than 75% of eligible members, as it had been in earlier years, reflecting
a continuing commitment to the project by most partners. Only once did
attendance fall significantly below this level during the whole project.
The external project evaluator was at a number of the meetings this year.
Meetings alternated between Southampton and other partner sites throughout
the project, taking the group to Cambridge, London, Nottingham and Oxford.
This strategy clearly helped maintain interest in meetings, spread the
organisational load (and costs) and alternately reduced travel requirements
for different members of the group.
A list of all project members is included
in Appendix 3.
From May 1997:
5 Year 3 Project
Or see the full list of project publications
Hitchcock, S, Carr, L, Harris, S, Hey, J M N, and
Hall, W, 1997, Citation Linking: Improving Access to Online Journals.
In Proceedings of the Second ACM International Conference on Digital
Libraries, Philadelphia, USA, July (ACM: New York), pp. 115-122
Hitchcock, S, Carr, L, and Hall, W, 1997,
Web Journals Publishing: a UK Perspective. Serials, 10(3), November,
Hitchcock, S, Kimberley, R, Carr, L, Harris, S,
and, Hall, W, 1998, Webs of Research: Putting the User in Control.
In Proceedings of IRISS'98: Internet Research and Information for Social
Scientists, Bristol, UK, March http://sosig.ac.uk/iriss/papers/paper42.htm
Hitchcock, S, and Carr, L, 1998, Linking Quality
Information Resources on the Web. A workshop presented at the UK Serials
Group annual conference, Exeter, UK, March/April. Slides
forthcoming on the Web
Probets, S, Brailsford, D F, Carr, L, and Hall, W,
1998, Dynamic Link Inclusion in Online PDF Journals. In Proceedings
of EP'98, the seventh International Conference on Electronic Publishing,
Document Manipulation and Typography, St Malo, France, April
Carr, L, Davis, H C, De Roure, D, and Hall, W,
1998, Application-Independent Link Processing. In Proceedings of the
Seventh International World Wide Web Conference, Brisbane, Australia,
April. Subsequently: Computer Networks and ISDN Systems, 30(1-7)
(frames version, Proceedings home page) or http://www.elsevier.nl:80/cas/tree/store/comnet/free/www7/1883/com1883.htm
(no-frames, paper only) or http://journals.ecs.soton.ac.uk/~lac/www7/257veryshort.html
Carr, L A, De Roure, D, Hall, W, and Hill, G, 1998,
Implementing an Open Link Service for the World-Wide Web. World Wide
Web, 1(2), 61-71
Carr, L A, Hall, W, and Hitchcock, S, 1998, Link
Services or Link Agents? In Proceedings of the Ninth ACM Conference
on Hypertext, Pittsburgh, USA, June (ACM: New York)
Hitchcock, S, Quek, F, Carr, L, Hall, W, Witbrock,
A, and Tarr, I, 1998, Towards Universal Linking for Electronic Journals.
Serials Review, to be published (an
updated version of Linking Everything to Everything: Journal Publishing
Myth or Reality? Presented at ICCC/IFIP conference on Electronic Publishing
‘97: New Models and Opportunities, Canterbury, UK, April 1997) http://journals.ecs.soton.ac.uk/IFIP-ICCC97.html
Hitchcock, S, et
al., 1998, Linking electronic journals: lessons from the Open
Journal project. Presented at Making the Most of E-journals, a seminar
organised jointly by UKOLUG and the UK Serials Group, Loughborough University,
UK, April, to be published
Overall project review (1995-98)
1 Project Review:
In contrast to the yearly reports, this section takes a broader perspective
on the Open Journal project, its achievements - successes and failures
- and discusses future prospects for the components, applications and link
publishing strategies. In addressing these issues, the section is structured
to enable those familiar and unfamiliar with the project alike to identify
key points which might stimulate further investigation. This report can
be read in conjunction with a separate, more detailed paper on lessons
from the Open Journal project , or with the three
annual reports to eLib, the last of which is included above.
The section is aimed at those publishers and Web information providers
which might in the future want to build stronger capability for linking
into their services. This might include developers and partners involved
in the project because, although the project did not develop a commercial
exit strategy, a number of initiatives have been set up between the partners
to explore specific requirements with a view to building tools and applications
that work in a commercial environment.
Links are an essential element of the Web, but link publishing - the
ability to manage link information separately from content - remains an
esoteric approach. Yet the first signs of commercial adopters outside the
community that developed the techniques can now be seen. Setting aside
the exact tools that might be used, and without specifying a timescale,
it can be speculated that link publishing will become common practice in
Web publishing applications.
It is not too late to investigate link publishing. Publishers involved
in the project already have a head start. This is not an unreasonable return
for participation, but it is stronger for two unplanned outcomes. First,
instead of using a general-purpose tool set, publishers now seeking to
build applications with the project developers are specifying custom software
based on knowledge and experience of the project demonstrators and of their
users' needs. Second, the project has no demonstrators that it can maintain
publicly, so for those outside the project and who missed the earlier open
demonstrations, it may be harder to understand what is happening.
For publishers the first sign that link publishing is emerging may be
a new service introduced by a competitor. That threat became real just
recently. As well as those publishers involved in this project, this report
reveals another commercial journal publisher which is building its own
link publishing technology and has announced a new development with possibly
profound implications for other Web journal publishers.
Highlights of the project
Demonstrated the value of citation links
Discovered user preferences for linking
Informed design of research version of link service based on user feedback
Built new software components specifically for e-journal applications
Began applying ideas and results to publisher-specific projects
Below are listed the objectives and deliverables outlined in the original
Performance of the project against original proposal
Against this measure the project was a success, with a significant proviso.
A 'framework' was developed, used to demonstrate 'Open Journals' and run
trials with publishers and users, and dissemination of the framework has
begun too, if in a more limited form than envisaged.
To develop a framework of information retrieval technologies and electronic
publishing practises to be used by information providers which will allow
them to make their publications available not as isolated, one-off resources,
but as co-operating assets within an information delivery environment such
as a library at an institution of higher education.
To develop agent technology for use within the framework to allow information
to be negotiated automatically from a variety of on-line database resources.
To use the framework to develop a set of ‘open journals’ based on existing
To run trials of the framework with publishers to determine its usability.
To run trials of the resultant journals with users.
To disseminate the framework and documentation to interested parties.
What most limits the objectives is not the technology, but concerns
the phrase 'co-operating assets'. The assets of journal publishers are
still some way from cooperating, and the ability of software agents to
'negotiate' automatically with online databases must be preceded by negotiation
between the content owners. Recognising this, the broader vision has been
set aside for post-project exploitation in single publisher environments
where negotiation between publishers is unnecessary, or in journal aggregation
models where agreements on promoting assets are already in place.
Two possible developments may hasten wider moves towards cooperating
assets as far as journals are concerned. First, shifts in the balance of
ownership of content from publishers to authors and their institutions,
stimulated by new publishing approaches such as e-prints, and use of these
contents in multiple media, may prompt moves to widen ways in which new
partnerships can cooperate. Second, there is evidence, perhaps indistinct
at present, that the Web is evolving from a simple distribution network,
a network of connected computers, towards distributed information and providing
more support for 'interactive' information resources.
a A Web-based distribution system for various journals’ contents,
with a document management system allowing various methods of access to
each document (by author, keyword, similarity of content).
Developments in electronic publishing
have rendered some of the proposed deliverables irrelevant. Almost universally,
publishers now distribute their journals on the Web, formatted in Adobe's
PDF, a process which began the year following the project launch and which
has been tracked and reported by the project. What has been delivered is
not a set of link databases for individual journals - linkbases were created
for Open Journals collectively rather than the contributing journals -
but a collection of software tools with which single journal linkbases
can be built as required. A flexible, fast and effective bibliography agent
has been demonstrated and can be applied as a standalone tool or in conjunction
with the tool used to build linkbases. Producing documentation for these
tools depends on improving the stability of the tools, on the range of
applications, and on progress towards commercialisation.
b A set of electronic journals, accessed through (a), formatted
using Adobe Acrobat or other rendering software as appropriate.
c A set of link databases for each of the journals in (b) to
link the text and pictures in each article to relevant subject-specific
information (e.g. other pictures, articles, database entries, network journals).
Each will be a ‘published package’ specific to the articles of a particular
d A set of ‘agents’ which can be used to determine relevant information
held in network-accessible databases. These agents will be configurable
and hence usable for other open publications in different subject areas.
e A ‘bibliography agent’ which, based on a set of rules and a
combination of typographic features and punctuation, can interpret the
bibliography entries used by various journals and return the referenced
article from an existing on-line source.
f Comprehensive documentation covering
(i) the use of the above software to produce an ‘Open Journal’
(ii) recommendations and guidelines based on the experience of converting
publications from print-based media into ‘network information assets’.
2 An Open Journal
framework: a concise summary
Tavistock suggests these issues: technical, organisational, economic,
scholarly, learning and teaching
The Open Journal approach
In practical terms the Open Journal approach involved, respectively, linking,
resources, formats and location:
In this approach the link service has been integrated with two new components
developed in the project:
applying new linking tools based on a 'link service', the Distributed Link
... to selected primary journals and secondary resources
... in HTML and PDF (and prospectively the Extensible Markup Language,
... distributed on the Web
a bibliography agent, to link primary and secondary resources via citation
a PDF module, to add links to documents in that format
Demonstrators and linking
Three demonstrator Open Journals were produced in the subject areas of
cognitive science, biology and computer science. These areas and the core
journal resources contained within the respective Open Journals were effectively
defined by the interests of the developer teams and of the twelve or so
publishers that supported the project (Appendix
2). It is more convenient, however, to consider the demonstrator Open Journals
from the viewpoint of the linking features they highlighted (Table 3).
The three main link types in this application can best be described as
citation links, keyword links and PDF links.
Citation linking. Citation links act on references contained within papers.
Keyword linking. Links articles containing a given keyword to other articles
for which that keyword was listed, by author or editor.
PDF linking. Supports linking, citation or keyword linking, for documents
Table 3. Open Journal demonstrators
and linking features
||Open release; closed end May 1998
||Released to selected evaluators
||Internal project release
User responses were gathered in a number of ways: meetings with individual
users, selected evaluation groups and an open access demonstrator, mediated
by electronic mail and Web forms. Typically, evaluators included specialists
in the subject areas covered, and included researchers, teachers, librarians
and publishers. Below are a selection of comments returned via a Web questionnaire
in response to the open demonstration of citation linking in the Open Journal
of Cognitive Science. The test results are presented
in full in Appendix 1a.
The user view
"It's a great service!"
"The potential is clear and exciting. ... Thanks for letting us see how
the future might look!"
"An excellent way to trace ideas and how the scientific community has reacted
"the forward search aspect is terrific."
"I'd like to have such a link structure for EVERYTHING I read."
"Looks like a wonderful way to find info that sometimes is elusive using
keyword searches of databases."
"It is a WONDERFUL idea. However..."
"It is a WONDERFUL idea. However, (for) two separate target BBS
articles ... it simply didn't work, and was slow doing it."
"I think this sort of thing is definite way to go, hope it isn't killed
by premature beta releases..."
"Powerful tool which better exploits the capability of the medium ... (but)
not fully implemented."
"I could have obtained the same information (and in a more controlled way
since I would have been deciding where and when to search) if I simply
had one window looking at an item and had another window open into BIDS/ISI
and used the citation search feature."
"The search engine needs to be improved before a serious trial can be initiated.
The single keyword restriction made the output set way too big."
"It would be a good idea to have an opportunity for marking citations."
Important results from user testing
Users are very demanding.
"It would have to be much faster."
"Speed of links - initial link to an abstract, have all the abstracts loaded
with the initial page--one can do something else while the article loads,
but once in "reading mode" 60-second time-outs are frustrating."
Citation links are popular, but users will always want more links.
Before any such services existed there was great anticipation among
e-journal advocates. By the time the project was, briefly, able to release
its demonstrator openly, real users were more sanguine. Why so few links?
(It was due to limited datasets.) The demand for links in this context
will establish very rapidly.
"Bigger and better please."
"Make it BIGGER!!!"
"I wish that a visual cue could be use to discriminate between links to
the reference section of the article alone, and those citations for which
an abstract could be found. I was a bit frustrated by wanting to look up
an abstract, interrupting my reading to find it, only to learn that it
"One could imagine even making the link go directly to the abstract/full-text,
rather than first to the list of references at the end of the paper."
"Add a tiny little document icon next to those links for which there are
abstracts or full-text available, and make the little icon jump directly
to the abstract/full-text. "
Technology must be transparent: users want better services without having
to install new software or change computer settings.
It was common practice for Web users, especially back in 1995 when
the project began, to download and install software from the Web to improve
the functionality of the Web experience. With the exception of Acrobat
it seems the practice of software download does not apply to typical e-journal
users, as the project soon discovered from publishers and librarians. So
the link service software was rewritten to work at the server end, mediated
by a user-set browser proxy. Even this was insufficient. Libraries do not
want settings on shared machines to be altered, and proxy settings can
interfere with firewalls in corporate environments. In the latest version
the direction to the proxy server is attached to the URL, leaving the user
to browse the Web conventionally and do nothing to receive the link service,
bar starting from the right place!
"I don't know what 'the proxy' is."
Links help navigation but users also need orientation.
The interface is the user's starting point with a journal, publisher
or collection of journals. Web interface design has advanced enormously
in the last year or so, especially for large resources, where so-called
'gateways' have leapt in importance. An Open Journal, or collection of
journals, needs not just links, but a distinctive entrance too.
"It was not easy to discover how to start into the archive."
"Took a while to see what was on offer."
"Without additional navigation aids it is easy to lose track of where you
are in the cluster."
Reliability is a critical issue.
"Unable to access. Multiple attempts over several days failed to connect
to linked server."
Links must be clearly labelled and unambiguous: users are suspicious
of unexpected links in texts
This section can be used to get a rapid overview of the project by those
who may wish to investigate future applications.
An Open Journal 'FAQ'
Has the project produced anything I can use?
Not the Open Journals of the title, because these were built as demonstrators
and are now closed, but the collection of software tools used to build
them may be available by arrangement.
What did the Open Journals demonstrate?
That a collection of information resources, in this case journals but
could be other types of resource, available over the Web but not necessarily
located in the same place or owned by a single source, could be independently
joined, or integrated, by the application of hypertext links.
Why would anyone want to do this?
Links are useful to readers because they provide access to new or additional
information. Publishers can add value by providing users with links, but
new services are needed to create and maintain links for very large collections.
Few publishers own sufficient data in any single subject to satisfy users,
so comprehensive linking strategies require different publishers to cooperate.
In principle, the simplest arrangement and lowest overhead for cooperation
between publishers would avoid data transfer and multiple sources, but
would allow quality links to be added by a separate and independent third-party
process. No publisher currently owns a 'gateway' to all journal resources.
Unless that happens - and it may appear unlikely - to extend and promote
access to its materials a publisher will increasingly need to provide links
from other providers' services.
How are these third-party links built?
These are not conventional HTML links, but are added to, or superimposed
on, pre-authored documents as they are served across the Web by software
called a link service. Additional software components have been built for
journal applications to automatically match references in published papers
with a database of abstracts.
What document formats can I work with?
The service works with HTML documents served on the Web.
Can it work with PDF?
Yes, the service can also be applied to PDF as well, but this is a
more complex application.
What software is available?
The Distributed Link Service, optionally with the PDF module or agent
software for making citation links. This is still research software which
may be commercialised. Availability for trial development is by arrangement
with the developers.
Can anyone build an Open Journal?
Not yet. Application requires programming skills. Current development
effort suggests that the approach is best employed for large-scale resources.
PDF is especially difficult to manage and demands powerful computing facilities.
Who might use this software?
With a more stable release of the software, a better interface for
developers, and better access to resources, this approach might eventually
be used by authors, librarians, or anyone building Web resources. But not
Who is using it?
A number of publishers involved in the project are planning to develop
their own journal linking applications using different combinations of
the software tools. These applications might become available as demonstrators
or as part of commercial services, and they might motivate commercialisation
of new versions of the software tools.
Does it work?
Yes, but it is not yet robust or stable enough for commercial application,
as can be judged from the results of the user tests. Informed by these
tests it is expected that many of the limitations will be resolved by the
more focussed post-project applications being built with publishers.
What can I do?
To find out more, use the list of contacts
at the end of this report.
by Chris Bell, the project's external evaluator
of the project
Aims and methodology of external evaluation
The aim of the external evaluation, comprising a total of some eleven days
work over the lifetime of the Project, has been to present a detached,
external and critical view as to the success or otherwise of the Project’s
development and process in order to improve its operation and delivery:
To achieve these objectives, the style adopted has been a formative and
illuminative, where the evaluation is approached in as open a way as possible
with key issues being derived through discussion, observation and analysis.
Issues thus derived have been discussed fully with the Project Director
and with other staff as appropriate in order to influence the direction
and progress of the Project positively. In addition, one interim report,
and mini-reports after each meeting, have been produced. These have highlighted
issues for consideration and have reviewed progress. Feedback has indicated
that issues raised have been considered and, where pertinent, generally
implemented by the Director and/or other staff.
To assess whether or not the aims and objectives of the Project are being
met in practice, and to recommend changes where necessary and practical
within the Project’s constraints.
To produce and discuss suggestions and recommendations regarding the Project's
strategy, management, operation and delivery, particularly relating to
user needs and views.
To produce and discuss evidence to inform reporting to eLib.
To produce and discuss evidence to inform Project continuation/exit strategies.
To provide support to Project staff in developing and undertaking their
own internal evaluation procedures.
Evaluation has included an analysis of strategic goals and planning,
management, operation and delivery. Data has been collected through
discussions with key players (face to face and via telephone), attendance
at Project Steering Group meetings, attendance at an all-day seminar, analysis
of documentation, and administration of a simple questionnaire to key players.
The external evaluation has not focussed upon the use, usability or
appropriateness of specific products or software; this has been the province
of Project staff and is noted elsewhere in this final report.
The Project’s overall aim ("To build a framework for commercial publishing
applications which enables online journals to be interlinked, and to provide
users with the ability to create or follow numerous flexible linked paths
designed to support themed study and research using the maximum available
online resources") has been achieved with aplomb. Measured against this
aim, the Project has been successful.
The Project has developed an innovative “product” which, if fully implemented
in a user-friendly and stable form, should be capable of satisfying real
needs of publishers and other end-users. It undoubtedly has the potential,
not yet fully realised, to satisfy one of the major strategic goals of
the eLib programme: to shape and accelerate the development and uptake
of electronic media and network services throughout higher education. In
addition, there are a number of potential applications in other areas,
including ICT-based teaching and learning, as a “front-end” for informational
databases and as an integrative product for multi- and mixed- media applications.
The Project has satisfied eLib intentions in its experimental, open-ended
and innovative nature, in its involvement of different key-players, and
in its potential to create conditions of sustainability. There is still
development necessary in the latter area. It has also clearly influenced
others working in similar fields, which is to be commended. There appears
no product with a similar range of functionality on the market at present
although during the three years of the Project, developments have taken
place, sometimes faster and with greater commercial focus, in several of
the Project’s areas.
Target end-users will undoubtedly benefit from the Project if its outcomes
are adequately exploited. To achieve this exploitation, there should
be continued development towards a stable and appropriate end-user product
(or products) for which funding will be necessary, either from grant or
The initial expectations of the work have been met (and surpassed in the
case of Web proxy server and citation linking).
The Project has brought together two leading-edge university departments
and a number of publishers into a cooperative consortium. Networking across
these organisations would not generally take place and is likely to lead
to further collaboration.
Demonstration of the Open Journal concept, and pilot implementation of
the concept in three areas, has been successfully achieved, raising an
awareness of new approaches to publishing and authoring and setting the
scene for further developments, some of which may be outside the Open Journal
Demonstration of the applicability of creating databases of links separate
from a multimedia or text database has been successfully achieved. This
has implications for other applications in addition to the Open Journal
Development of software to insert generic links into PDF materials and
showing that PDF can be dynamically expanded with added value annotations.
Dissemination via papers, conferences, Web and individual contact has been
successful, increasing the overall pool of knowledge in the areas of the
work and opening a number of possibilities for future development
Issues from which to learn
An initial, and on-going, underestimation of time it would take to develop
a successful implementation of the Open Journal concept has led to less
than optimal end-user trials and some tension between developers and publishers.
The concepts of the work are innovative (and probably “ahead of their time”),
requiring a change to the way people think and work. The time necessary
to achieve this change and acceptance, and to “get the message across”
to publishers and other potential users, has been underestimated. Greater
consideration of the needs, working styles and psychology of end users
and intermediaries may have been appropriate.
There have been some difficulties, particularly earlier in the Project,
about the apparently divergent aims, needs and working styles of the universities
and the publishers (e.g. an “academic” approach versus a “commercial” approach,
meeting deadlines). Increased two-way communication about fundamental aims,
intentions, milestones and achievements would help resolve these difficulties.
Developments and changes along the way have resulted in software which
is complex and more difficult to use than necessary. Upon reflection, an
approach where reliable and easy to use “core” software was developed,
with add-ons for specific purposes, may have been more appropriate.
For commercial exploitation (e.g. publishers to implement the software)
there needs to be a major rewrite informed by the work of the Project.
Although there still appear to be no products on the market with similar
capabilities, other products which have similar intentions have been developed
and are being adopted by some publishing houses.
There is a need to identify and communicate better how the Project differs,
and adds to, other publishing initiatives.
Continued collaboration between the two Universities and between the publishers
Continued development of the work, in particular metadata developments;
further research and development funding will be necessary.
Possibilities of new projects led by publishers, building upon work of
the Project and drawing upon some of the software developed; however there
are issues of IPR to be resolved.
Development of the “back-room” software tools produced by the Project into
Process lessons learned for future projects
Ensure quality communications and open dialogue between all key players
with adequate face-to face meetings (particularly near the start of a project).
Encourage ownership and a shared understanding of the benefits (and limitations)
of the partnership.
Undertake periodic review (and re-statement where necessary) of a project’s
vision, aims, timescales, milestones, deliverables, dissemination and exploitation
Ensure adequate quality inter-project collaboration facilitated by the
The route(s) to market, project continuation and exploitation strategies,
and issues of IPR ownership, need to be considered early, continually and
Ensure adequate end-user evaluation of products as they are developed in
order to help ensure optimal usability and appropriateness and to encourage
University of Plymouth
4 Overall conclusions
What has been gained
First major user tests of an emerging publishing technology
First large-scale demonstrations of citation linking
Informed by user tests, produced some successful, original software
Publishers can see the value of linked citations, even if they can't see
how to do it!
Publishers are interested in the PDF rewriting capability.
No agreements to preserve legacy demonstrator Open Journals on the Web.
Took on too much: developing software and attempting three application
scenarios, only two of which were developed far enough to be evaluated
Software was too diverse: trying to develop and maintain an in-house Web
server and proxy, which was not a core part of the project, as well as
develop linking, formatting and agent software.
Did not involve publishers sufficiently. Few publishers identified with
the demonstrator Open Journals. Publisher-specific post-project work is
being embraced with more interest, focussing on real publisher problems
and needs, and producing some surprising initiatives.
Could we have produced better Open Journals? Towards
a journals 'gateway'
Yes, the demonstrator Open Journals could have been better. Mostly this
answer is informed by results, as a research and development project should
be, but other elements might have been foreseen. The Open Journals would
have been better if we had:
Although Open Journals are a way of aggregating large resources through
linking, a complementary approach which was not fully developed would
have been to build the Open Journals as gateways, of the type that have
thrived generally on the Web and elsewhere in eLib. One way of exploring
this would have been to work with a subject-based information gateway project
within eLib to combine technologies. There were attempts to do this but
not enough time to develop the planned demonstrators. Of more relevance
to publishers, however, would be a gateway focussed on journal resources,
with link mediation provided by the Open Journal software.
access to more resources
built better user interfaces
stable, open systems technology for the Web, links, document formats
more link editing tools
attracted more users, sooner in the work
Publishers: the way forward
Since 1995, when the project began, the number of e-journals available
via the Web has jumped from around 100 measurably towards 3000, with perhaps
more than 5000 worldwide according to some speculation (Table 4). Earlier
estimates indicate that more than 80 per cent of e-journals are presented
in PDF, which is the de facto standard for e-journals reproduced
from print. It is against this background that the project's link application
must be considered.
Table 4. The pace of change: summary of the
growth of e-journals since 1995
et al., Association of Research Libraries
* Some predict
we will reach 5000+ e-journals, worldwide, during 1998
||c. 1300 'UK' e-journals
et al., Serials 10(3), 285-299
Recently, one publisher reportedly described the growth of PDF e-journals
as the 'first frontier' facing journal publishers just a couple of years
ago. The second frontier is the emergence of links, he went on to say.
It has not happened yet, but a path may be being mapped out in post-project
applications with publishers. In turn, the success of these publisher applications
may determine whether the link service software, or which elements of the
software, are developed as commercial tools.
Post-project plans of participating publishers
Three publisher applications have begun, or will be continue to be investigated,
post-project. The dilemma for these applications, whether to continue with
a static format such as PDF or to migrate to a more flexible format for
Web publishing, is highlighted by two contrasting observations of the project
Clearly the latter has a bearing on immediate applications because it reflects
existing publisher cultures and practices. Most of the continuing publisher
projects involve working with PDF. These projects are supported by
the original project developers but with a view to the publishers taking
control and applying the tools themselves. All three approaches have one
thing in common: they do not yet plan to use the full capability of the
link service but instead will initially use a cut-down version of the tools:
PDF proved to be quite a complicated format for the project to work with.
The format is optimised for display, links are specified with respect to
page coordinates not to words or phrases, and in-depth knowledge of the
PDF file format is required to access and create the necessary structures
to add links around words and phrases.
Experienced Webmasters in established publishing environments are reticent
about using databases of external links as a fundamental navigation resource.
Instead they prefer to produce pre-linked documents from bespoke database
In addition there is the ongoing work with the JISC-funded Nature
digitisation project to build link trails in a large PDF archive.
One will use the text recognition properties of the bibliography agent
to extract citation data from PDF journals.
Two will use the agent technology to match extracted citations within their
own data sources.
One is using the link service to insert links in static pre-processed PDF
This all suggests that e-journals are not yet ready to become distributed,
dynamically changing resources in a native Web format rather than PDF.
However, this could change quickly. Publishers, now more familiar with
working in multiple media, are attracted to the reusability features of
the SGML format. Journal publishers in particular are formatting selected
elements of published papers, notably header data, in SGML, and some have
plans to generate HTML for the Web from an SGML original.
First examples of reformatting to support linking include conversion
of reference sections to HTML, as demonstrated by the Institute of Physics
e-journals service. This service, which uses PDF presentation, links citations
from extracted HTML to a database of abstracts held by the publisher.
Alternatively, publishers may be tempted towards the Web based successor
to SGML, XML, if it delivers more cost-effective production, particularly
if e-journals can generate their own independent income streams to support
this development. XML, and its linking component, potentially offers significantly
more native capability for linking applications than does PDF, or even
HTML, but it is not yet widely used. XML became a recommended standard
only six months ago, early in 1998, and the linking components are not
yet included in that standard. As a possible indicator to the nature of
XML linking, research which pre-dated the project and which produced the
link service software used in the project, was cited as an early influence
on the standard.
There is belated recognition that e-journals must offer more than the
printed equivalent. Citation linking will be the first example, and commercial
agreements have been signed to do this in principle but not yet in practice.
There are a number of possible effects. As more data is shared, how will
it be managed, by whom and where? As shared data sources become larger,
will static linking be adequate in a fast-changing, expanding data environment?
A platform for commercial tools
A version of the link service software is available from Multicosm
Ltd, although it does not currently support journal applications as
developed in the project. Negotiation continues with the company with a
view to commercialising the link service for publishers, possibly with
the additional components built in the project. This process will be informed
by demand from publishers, particularly those experimenting with their
The final word: pushing back the boundaries
Although the application of project technology was not without its difficulties,
and the demonstrators need to be followed up by real applications, a bigger
barrier to adoption of an Open Journal approach remains cultural. The cultural
shift required to embrace it is perhaps best highlighted by a description
of the advantages of using a link service:
Allowing others to exercise editorial control over already-published materials
would appear to run counter to publishing ethos. This may be one reason
why project publishers appear reluctant to unleash the full link service
on their Web journals in the follow-up projects. Internationally there
are political and legal moves to strengthen the exercise of rights and
control over data published on computer networks. Yet nobody can know the
impact that universal adoption of the Web, a uniquely user driven service,
will have as a communications medium.
Although the link service approach may seem over-complicated, an advantage
is that links can be applied directly to citations in any documents, from
any publisher, not just those over which the user has editorial control.
It is possible that anarchic users will stretch the limits of what is
acceptable, but the motivations for change can be seen even among established
publishers, who invariably have limited access to Web users. What if a
publisher could extend its reach by allowing direct links to its works
to appear in other services, library services for example? How could these
links be maintained, updated and managed? Is it possible that respected
publishers might want to do this, interact with other services?
One is. The Institute of Physics' Stacks service - 'the ultimate linking
service' - generates tables of contents (TOCs) with embedded hyperlinks,
and is aimed at librarians, other publishers, aggregators, abstracting
and indexing services and producers of information gateways. In contrast
to the project's link service, Stacks delivers TOC and link data via email
or file transfer to the local service provider.
Is this a more practical approach than the project has applied, more
likely to appeal to publisher needs, or is it simply more limited and less
flexible? Whichever, an important principle has been recognised by two
developments independently: data, not just computers, are becoming perpetually
more distributed on the Web. No data provider can survive alone. Data will
be shared and interactive, and not just at the user level. The sooner this
is more widely recognised, the more likely that established cultures can
begin to change and efforts can be directed towards building an information
environment in which new opportunities to serve users can flourish, rather
than trying to constrain this environment by recreating and imposing other
The legacy of the Open Journal project may eventually be commercial
applications built by publishers and supported by commercial tools first
tested in the project. Perhaps a broader legacy will be to have influenced
developments leading towards distributed data, by motivating the user benefits
at a time when the prevailing culture, especially among information providers,
was difficult to reconcile with the emerging needs.
Carr, L A, De Roure, D, Hall, W, and Hill, G, 1998, Implementing
an Open Link Service for the World-Wide Web. World Wide Web, 1(2),
5 Open Journal
Project Publications: the Complete List (by author)
Carr, L A, Hall, W, and Hitchcock, S, 1998, Link Services or
Link Agents? In Proceedings of the Ninth ACM Conference on Hypertext,
Pittsburgh, USA, June (ACM: New York) pp. http://diana.ecs.soton.ac.uk/~lac/imp.pdf
Carr, L, Davis, H, De Roure, D, Hall, W, and Hill, G, 1996, Open
information services. Computer Networks and ISDN Systems (special
issue, Proceedings of the Fifth International WWW Conference), Vol. 28,
Carr, L, Davis, H C, De Roure, D, and Hall, W, 1998, Application-Independent
Link Processing. In Proceedings of the Seventh International World Wide
Web Conference, Brisbane, Australia, April subsequently Computer
Networks and ISDN Systems, 30(1-7) http://www.elsevier.nl:80/cas/tree/store/comnet/free/www7/00/index.htm
(frames version, Proceedings home page) or http://www.elsevier.nl:80/cas/tree/store/comnet/free/www7/1883/com1883.htm
(non-frames, paper only) or http://journals.ecs.soton.ac.uk/~lac/www7/257veryshort.html
Carr, L, Davis, H, Hall, W, and Hey, J, 1996, Using the World
Wide Web as an Electronic Library. Third international ELVIRA conference,
Milton Keynes, May http://diana.ecs.soton.ac.uk/~lac/elvira-full.html
Carr, L, De Roure, D, Hall, W, and Hill, G, 1995, The Distributed
Link Service: a Tool for Publishers, Authors and Readers. World Wide
Web Journal (special issue, Proceedings of the Fourth International
WWW Conference) No. 1, Winter 1995/96 http://www.w3.org/pub/Conferences/WWW4/Papers/178/
Hitchcock, S, et al., 1998,
Linking Electronic Journals: Lessons from the Open Journal Project. Presented
at Making the Most of E-journals, a seminar organised jointly by
UKOLUG and the UK Serials Group, Loughborough University, UK, April, to
Hitchcock, S, 1996, Web publishing: speed changes everything.
IEEE Computer, Vol. 29, No. 8, August, 91-93
Hitchcock, S, and Carr, L, 1998, Linking Quality Information
Resources on the Web. A workshop presented at the UK Serials Group annual
conference, Exeter, UK, March/April. Slides forthcoming
on the Web
Hitchcock, S, Carr, L, and Hall, W, 1996, A Survey of STM Online
Journals 1990-95: the Calm Before the Storm. In Directory of Electronic
Journals, Newsletters and Academic Discussion Lists, sixth edition,
edited by D. Mogge, (Washington, D.C.: Association of Research Libraries),
pp. 7-32, http://journals.ecs.soton.ac.uk/survey/survey.html
Hitchcock, S, Carr, L, and Hall, W, 1997, Web Journals Publishing:
a UK Perspective. Serials, 10(3) November, 285-299 http://journals.ecs.soton.ac.uk/uksg.htm
Hitchcock, S, Carr, L, Harris, S, Hey, J M N, and Hall, W, 1997,
Citation Linking: Improving Access to Online Journals. In Proceedings
of the Second ACM International Conference on Digital Libraries, Philadelphia,
USA, July (ACM: New York), pp. 115-122 http://journals.ecs.soton.ac.uk/acmdl97.htm
Hitchcock, S, Kimberley, R, Carr, L, Harris, S, and, Hall, W,
1998, Webs of Research: Putting the User in Control. In Proceedings
of IRISS'98: Internet Research and Information for Social Scientists,
Bristol, UK, March http://sosig.ac.uk/iriss/papers/paper42.htm
Hitchcock, S, Quek, F, Carr, L, Hall, W, Witbrock, A, and Tarr, I,
1998, Towards Universal Linking for Electronic Journals. Serials Review,
to be published (an updated version of Linking
Everything to Everything: Journal Publishing Myth or Reality? Presented
at ICCC/IFIP conference on Electronic Publishing ‘97: New Models and
Opportunities, Canterbury,UK, April 1997)
Hitchcock, S, Quek, F, Carr, L, Hall, W, Witbrock, A, and Tarr, I,
1997, Linking everything to everything: journal publishing myth or reality?
ICCC/IFIP Conference on Electronic Publishing, Canterbury, UK, April
Open Journal Project, 1997, Can links become the new publishing
model for the Web? The Open Journal approach. A compilation of conference
Probets, S, Brailsford, D F, Carr, L, and Hall, W, 1998, Dynamic
Link Inclusion in Online PDF Journals. In Proceedings of EP'98, the
seventh International Conference on Electronic Publishing, Document Manipulation
and Typography, St Malo, France, April
Probets, S, 1997, PDF, databases and the Internet. Acrobatics,
2(1), March, 3-7
For a classified and more up-to-date list of the complete publications
see the project's Papers
Appendix 1a. User results:
Open Journal of Cognitive Science, version 2.1
This open release followed two earlier releases to small groups of subject-specialist
evaluators. Results from the first release were reported internally within
the project and in the second annual report to eLib, but were not posted
publicly. The second release was not reported, but resulted in modifications
for version 2.1.
The principal changes in version 2.0 were an improved layout, other
new presentational features, and speed of link serving. Version 2.1 saw
the proxy setting added explicitly to the url, rather than requiring the
browser settings to be changed.
Version 2.1 was notified to several large listservs and other mailing
lists, and was publicly available from 28 April to 31 May 1998. Users
were asked to complete a Web form, and 21 replies were received between
1-21 May. It is not possible to tell which mailings prompted most replies,
nor to tell the backgrounds of the respondents.
The overall level of response was disappointing. Methods of communication
have an impact. Email forms appear to be less effective than Web forms.
There may not have been enough incentives for potential users. Otherwise
judgement on the relative response and its significance for the product
being evaluated must be subjective.
Due to an error, one of the notices was mailed with the incorrect URL
for the Web form, pointing those users instead to an earlier form. This
form corresponds closely with the final version, but not exactly. Results
shown in brackets below indicate replies submitted using the earlier form.
The results may not be statistically significant, but some observations
can be made:
Ironically, the results understate the number of link errors in this release,
which were discovered by simultaneous internal testing. There were two
principal sources of these errors. First, the late change to the proxy-in-the-URL,
which produced links that take users out of the Open Journal (effectively,
in this situation users see the journal data in its original, unlinked
form; it would not appear obviously wrong, but the Open Journal functionality
is irrevocably lost). Second, inconsistencies in the citation style of
the journal Behavioral and Brain Sciences which prevented the citation
agent from correctly matching and linking the different resources. Both
were tackled prior to release as far as deadlines would allow, but the
time available was inadequate.
Reliability of the link service has been improved from earlier releases.
Link presentation and navigation appears to be satisfactory, with some
important qualifications concerning speed and the important role of the
entry page, or home page, for the Open Journal.
The resource collection could be enhanced.
Some users made allowances because the release was a demonstrator, not
a commercial product.
Overall impact was mixed: some approve of the innovations; others were
wary, perhaps because of unexpected effects or incompleteness; others were
unclear what the innovation was.
Major requirements: faster, bigger (more content resources), better search.
Yes, the form would have been revised had there been another release
and would have been designed to offer more incentives, to better anticipate
the users' interests and concerns, and to find out more about the users.
Responses to the questionnaire
1. Set-up: Were you able to set-up and run the OJ successfully
given the release email and instructions on the Web? Yes
12 (5) No 0 (1) Unanswered 3 (0)
2. Presentation: This covers things like access to the
journal contents and search facilities
a. Was it easy to navigate the OJ structure? Yes 9 (5)
No 3 (1) Unanswered 3 (0)
b. Can you suggest improvements or additions to make the OJ easier
or more intuitive to use?
3. Resources: all the resources listed on the contents page
for the cognitive science OJ
No, seems fine.
Some form of graphic indicating the connectivity between the current item
being read and the other items in the cluster might be useful.
I wish that a visual cue could be used to discriminate between links to
the reference section of the article alone, and those citations for which
an abstract could be found. I was a bit frustrated by wanting to look up
an abstract, interrupting my reading to find it, only to learn that it
was unavailable. Did I miss something?
Add publication dates to each article....couldn't see them.
The search engine needs to be improved before a serious trial can be initiated.
The single keyword restriction made the output set way too big.
Faster and better search resources.
It would have to be much faster.
Unable to access. Multiple attempts over several days failed to connect
to linked server.
I would have liked to see papers citing the one I was reading (Geary in
BBS well-linked). Couldn't find a way to do this.
(More explicit icons and less text to read.)
(I didn't get started. It was not easy to discover how to start into the
archive. I would have appreciated some ready means of starting from something
like a journal table of contents.)
(Two simple suggestions: 1. There is apparently some way in HTML to change
what is displayed in the bottom status line as your cursor moves over a
link. The default is to display the address of the link; but I think it
would be very useful to display an abbreviated version of the full citation.
That way, you can actually get much of the information you need by just
passing your cursor over the link--you don't even need to click and jump
to the link. For many journal citations, the first, say, 8-10 words of
the title, and the rest of the citation would probably fit on one line.
In fact, if this works well enough, one could imagine even making the link
go directly to the abstract/full-text, rather than first to the list of
references at the end of the paper. 2. Don't use the default grey background,
for displaying article text, use something lighter like white.)
(Opening of new windows/browser views when following links so that the
original article is visible Just slightly less taxing on a weak memory!)
(Took a while to see what was on offer - my initial search by content yielded
relatively sparse links.)
a. Is this collection a useful basis from which to explore
the OJ concept? Yes 8 (4) No 1 (0) Unanswered 6 (2)
b. 'Wish list'. What online resources would you like to see
linked to this Open Journal? (recommendations of freely accessible
Web resources welcomed)
Without additional navigation aids it is easy to lose track of where you
are in the cluster.
Noetica: A Cognitive Science Forum http://psy.uq.edu.au/CogPsych/Noetica/
Is there someway to link to citations by some webcrawling technique say
to Medline? The forward search aspect is terrific.
Have you seen http://theory.lcs.mit.edu/~dmjones/hbp
(Other subject related journals.)
(Subject dictionary; basic texts; subject encyclopedias.)
(Link through to: one of the search engines; author's home page.)
a. Were you able to view and follow links served from the
link server? Yes 11 (5) No 1 (0) Unanswered 3 (1)
b. Were the added links useful? Yes 8 (4)
No 2 (1) Unanswered 5 (1)
c. Would you want to change the way the links are presented?
Yes 2 No 8 Unanswered 5
d. Did you encounter many broken links? Yes 1 (1)
No 10 (4) Unanswered 4 (1)
See Q2b: I wish that a visual cue could be used...
Another idea for link presentation: add a tiny little document icon next
to those links for which there are abstracts or full-text available, and
make the little icon jump directly to the abstract/full-text. Then, I don't
have to click the ref. to jump to the reference list to find out if there
is an abstract available; I can see directly in the text. Then the standard
ref. link can jump to the reference list at the end as it is now (but it
would still be nice to have the citation appear instead of the link address
in the bottom status bar, as I mentioned earlier).
More development ... most of the ones I used were not developed and finished-up
(I didn't get started.)
(Direct link from hypertext citation in article through to database (rather
than to bibliography)?)
If yes, please specify URL and highlighted link copy.
a. Was the service available when you wanted to use it?
Yes 12 (5) No 0 (0) Unanswered 3 (1)
b. Has the use of this service affected your non-Open Journal
work on the Web in any way?
Yes 2 No 9 Unanswered 4
If yes, how?
6. Your overall reaction:
Better accessibility of materials.
I told the editor of J.UCS (an electronic journal in Computer Science)
about your experiment.
a. Did the Open Journal match your expectations?
Yes 9 (3) No 2 (2) Unanswered 4 (1)
b. What did you most like and dislike about the OJ?
c. What changes would you make to the OJ to serve your needs?
I'm not clear as to the advantages of this approach. Many/most Web based
publications have links to their own bibliography and out onto the Web
if the cited item is available there. With regard to the citation information
I could have obtained the same information (and in a more controlled way
since I would have been deciding where and when to search) if I simply
had one window looking at an item and had another window open into BIDS/ISI
and used the citation search feature. It may, of course, be that I am missing
some important point here :-) . Or it may be I have missed some important
descriptive text that would have explained in more detail what advantages
there are in this approach.
Well, it is a very neat idea, but see Q2 for a frustration: I wish that
a visual cue could be used...
Best: ranking. No dislikes, yet.
An excellent way to trace ideas and how the scientific community has reacted
to them. I can only hope for continuation and expansion of this initiative.
Forward search is terrific.
There is nothing to dislike. If one simply wants to read an article without
looking at any citations, one can do that. It's no difference to the way
we are used to read. It is merely an additional, helpful feature.
It is a WONDERFUL idea. However, attempts to follow links in two separate
target BBS articles failed; I'd click and stare at blank screen for 1-2
minutes before finding myself at the top or bottom of the target article.
It simply didn't work, and was slow doing it (I'm on a campus Ethernet
connection, normally quite fast).
Like = powerful tool which better exploits the capability of the medium,
when fully implemented. Dislike = not fully implemented.
I'd like to have such a link structure for EVERYTHING I read...
Liked best the straightforward instructions; is very easy to use. Did not
like the long wait between clicks.
Easy to "hop" from one article to another, looks like a wonderful way to
find info that sometimes is elusive using keyword searches of databases.
Didn't see any "forward" references. I think that would be particularly
(The first search went fine. The next on BBS for 'amnesia' gave me three
articles, yellow on brown! and the copies of the full articles had no embedded
links or links on the citations at the end.)
(I couldn't get started easily enough.)
(It's a great service!)
(Unevenness of the links - some articles are good examples, others are
interesting in content but unsuitable for this demonstration.)
7. The way forward:
Add more journals.
Some of the interfaces need some work, e.g. search boxes should be next
to the search buttons (see Psycoloquy page).
It would be very useful to find out where an abstracted paper has been
cited. As far as I could tell, this info was only available for papers
cited in an abstracted paper and not for the abstracted paper itself.
Need expanded forward search transcending citation index.
Include possibility to access more abstracts via KEYWORDS.
Make it BIGGER!!! (Not a criticism, but something I hope for!)
(Clearer options, more links and more articles.)
(Make it easier to get started. (I should add that I am very computer-wise
-- the site did not impress me as user friendly and I had no motivation
to spend more than a few minutes seeing if I could get started.)
(Bigger and better please.)
What would be your major proposal(s) for any future release of this
This is comment on the Evaluation form. There needs to be more options
when asking about the usefulness of a feature. 'yes/no' does not enable
me to say 'sometimes'. A list of options like, 'always', 'often', 'sometimes',
'never' would be more useful.
To delay such a release as little as possible!
Develop webcrawling technique for citations.
1. I think it would be a good idea to have an opportunity for marking citations.
For example, I may be interested in some non-electronic articles which
are cited in the online article I am reading. It would be useful for me
to mark these citations and download or print them afterwards (perhaps
including the abstracts). 2. Just for curiosity: show the number of articles
which cite a given work at the top of the list.
Speed of links - initial link to an abstract, have all the abstracts loaded
with the initial page--one can do something else while the article loads,
but once in "reading mode" 60-second time-outs are frustrating. Then have
the abstracts contain links to entire article if someone really wants to
go on tangent. This was not a fair test of the system by any means; be
sure the software is working next time. NB: I realize not all the links
are 'live' - I tried a number, and my objection is to the time it took
to find out one was dead... I think this sort of thing is definite way
to go, hope it isn't killed by premature beta releases...
Encourage authors to reference other WWW resource, not only citeable literature,
in their articles.
(If it's for evaluation of the system, a smaller subset with more links.
If it's for actual use, more data.)
(Make it clear that only a subset of articles are suitable as demonstrations.
-- highlight them by use of colour in the contents tables? Thanks to the
team for letting us see how the future might look!)
The project is grateful to all those who gave their views on this and earlier
tests of other Open Journals and who contributed a vital element of the
Appendix 1b. User questionnaire
The Open Journal
Back to the Open Journal of Cognitive Science
|We hope you have had a chance to view
and explore this Open Journal. What struck you most about it? As you will
have noticed, it isn't a 'journal' as such, but a way of aggregating, or
linking, related resources in a way which ought to be useful for the reader.
To encourage your views there is attached below a short series of
The primary purpose of this evaluation is to inform further development
of this and other Open Journals. Your responses to the evaluation will
be formally reported to the project's developers, partners and publishers.
A summary report will also be forwarded to the directors of the Electronic
Libraries (eLib) programme, the body funding the project in the UK,
to inform their drive for integration between eLib projects where possible
and for sharing of information with other projects. A copy of the final
report to eLib will also be made available on the Web during the summer.
If you wish to be alerted when this report is available, please email Steve
Hitchcock, heading the mail 'Send final report alert'.
This is still a system under development. If you have had problems,
our apologies, but please do report them since this aspect of the evaluation
is as important as any other.
This framework covers issues such as set-up, presentation and the interface,
journals and other resources, links, reliability and your reactions and
thoughts on future work.
"Many thanks for your continuing help and support."
Appendix 2. Publishers involved
in the project
This is the complete list of publishers involved during the project:
For publisher links see the project's Partners
Bath Information and Data Services (BIDS)
British Computer Society
Cambridge University Press
Chapman & Hall
Company of Biologists
Stevan Harnad, Cognitive Science Centre, Southampton University
Institute for Scientific Information (ISI)
MCB University Press
Oxford University Press
John Wiley & Sons Ltd
Appendix 3. Project people
This is the complete list of people who were at some stage involved in
For contact details see the project's Personnel
Rosemary Altoft, publisher partner, John Wiley
Helen Atkins, publisher partner, ISI
David Barron, project chair, Southampton University
Chris Bell, project evaluator
David Brailsford, project director, Nottingham University
David Brown, publisher partner, BIDS
Les Carr, project manager, Southampton University
David Evans, research assistant, Nottingham University
Owen Garrett, publisher partner-technical support, Company of Biologists
Chris Gibson, publisher partner, Academic Press
Geeti Granger, publisher partner, John Wiley
Wendy Hall, project director, Southampton University
Stevan Harnad, publisher partner, Southampton University
Steve Harris, research assistant, Southampton University
Steve Hitchcock, research assistant, Southampton University
Rupert Hollom, research assistant, Southampton University
Ian Jones, publisher partner, British Computer Society
Robert Kimberley, publisher partner, ISI
Ed Pentz, publisher partner, Academic Press
Steve Probets, research assistant, Nottingham University
Freddie Quek, publisher partner, BioMedNet Ltd
Martin Richardson, publisher partner, Oxford University Press
Alan Robiette, FIGIT representative
Richard Skaer, publisher partner, Company of Biologists
Mike Stout, publisher partner, Oxford University Press
Anthony Watkinson, publisher partner, Chapman & Hall
Bryony Watson, publisher partner, Chapman & Hall
Peter Wilkinson, publisher partner, British Computer Society
Mathew Wills, publisher partner, MCB University Press
About the project; about this report
Steve Hitchcock firstname.lastname@example.org
About the link service software
Les Carr email@example.com
About PDF linking
Steve Probets sgp@Cs.Nott.AC.UK