Link Extinction

Extinction of links is also called “link death” or “link breaking”. A link that no longer works is called a broken link, dead link, or dangling link. Formally, this is a kind of hanging pointer - the object referenced by the link no longer exists.

One of the most common reasons for broken links to appear is that the webpage that the link points to no longer exists. This often leads to a 404 error, which indicates that the web server is responding, but it cannot find the specified page. Another kind of dead link occurs when the server that contains the page stops working or moves to another domain name . The browser may return a DNS error or display a site that is not related to the page you were looking for. The latter can happen if the domain name is transferred to another owner. Other reasons for broken links may include:

• The website has been rebuilt or redesigned, or the underlying technology has been changed, resulting in a large number of inbound and outbound links changing or becoming inaccessible.
• Many news sites store articles for a short time, and then translate them into . This leads to a significant loss of links in news discussion groups using informational Internet portals for links.
• Content may automatically become inaccessible after a certain period.
• Content may be intentionally deleted by the owner.
• The server may be updated and the code (for example, PHP) may not work as a result correctly.
• Links may be removed in a lawsuit.
• Search results from social networks such as Facebook and Tumblr are prone to broken links due to frequent changes in user privacy, account deletions, search result links to dynamic pages that give new results that are different from the cached result , or the removal of a link or photo.
• Links may contain short-lived, user-specific information, such as session or login date. Since such information is not true all the time, the result may be a broken link.
• The link may be broken due to some types of blocking, such as content filters or firewalls .
• A website can be closed or turned off, which leads to broken links if they point to this website.
• A website may change its domain name. Links to the old domain name can then become broken.
• Dead links can happen on the server side when content is collected from sources on the Internet without proper link checking.
• When new private gTLD domains become popular, top-level domains such as .mcdonalds or .xperia are discontinued ^[2] .

Answer 404 “Not Found” is familiar even to casual users on the network. A large number of studies examined the prevalence of broken links on the Internet, in the scientific literature and in electronic libraries ^[3] . In a 2003 experiment, Featherly (et al) found that approximately one in 200 links disappear from the Internet each week. McCone (et al., 2005) found that half of the URLs cited in articles were not available 10 years after publication, while other studies showed even worse link extinction in the scientific literature ^{[4] [5 ] ]} . Nelson and Allen ^[6] studied the extinction of links in digital libraries and found that about 3% of objects were inaccessible after one year. In 2014, Maciej Tseglovsky, owner of the bookmarking site, reported that a “fairly stable share” of 5% of links died out in a year ^[7] . Examining Yahoo! Links showed the half-life of a random page in 2016-2017 (shortly after Yahoo! stopped publishing this directory) for about two years ^[8] .

Some studies in the early stages of the Internet (in the late 1990s - early 2000s) showed a significant difference (more than an order of magnitude) of half-lives between different collections of links ^[9] .

In 2014, researchers at Harvard Law School Jonathan Zittrane, Kendra Albert, and Lawrence Lemming found that approximately 50% of the URLs in the US Supreme Court findings did not refer to the source information ^[1] . They also found that in the set of legal journals between 1999 and 2011, more than 70% of links did not work as expected. A 2013 study of magazine analyzed about 15,000 links in abstracts from the pages of Thomson Reuters and found that the average life time of web pages was 9.3 years, and 62% were archived ^[10] . In August 2015, the Weblock.io website analyzed over 180,000 links from the texts of the three main publishers with open access and found that more than 24.5% of the cited links are inaccessible ^[11] .

Detection of broken links can be done manually or automatically. Automated methods, including plugins for WordPress , Drupal and other content management systems , can be used to detect broken URLs . An alternative is to use broken link test tools such as Xenu's Link Sleuth . However, if the URL returns an HTTP code of 200 (OK) , the page may be accessible, but the contents of the page may be changed and is no longer relevant. So manual page checking, apparently, should be mandatory. Some servers return soft 404 , telling the requesting computer that the link is working, although, in fact, it does not work. Bar-Yosef (et al., 2004) ^[12] developed a heuristic algorithm that automatically detects pages that return soft 404.

There are many solutions to overcome broken links. Some methods try to prevent them altogether, while others try to get around them when a broken link is detected. There are also many tools to combat the extinction of links.

From the authors

• Carefully select and use hyperlinks and check them regularly after publication. The best technologies include links to the main sources, rather than secondary ones and preference should be given to sustainable sites. McCone et al in 2005 suggested avoiding quoting URLs that link to personal pages of researchers.
• Always look for the most compact and direct URL and make sure that it is a semantic URL without irrelevant information after the base URL ^[13] . This process is often referred to as URL normalization or .
• As far as possible, use persistent identifiers such as ARK ( Archival Resource Key ), DOIs , links, and PURL .
• Avoid links to PDF documents where possible, since PDF documents, after all, documents, not web pages, their contents may change without notice, and their names often contain characters, such as a space, so they need to be encoded for url. Large PDF documents can load slowly and cause a timeout error ^[13] .
• Avoid providing links to pages deep for the site, which is known as external linking .
• Use website archiving services (for example, WebCite ) to constantly archive and retrieve quoted Internet links ^[14] .

Server Side

• Never change URLs and never delete pages. If there are reasons why the page is no longer needed, such as editing a message on news sites, replace it with a page explaining the reasons for the removal.
• If the URL changes, use a redirect mechanism such as “301: Moved Permanently” to automatically inform browsers and search engines about the new location.
• A web content management system can provide embedded link management solutions by updating them if they change or move around the site.
• WordPress prevents link extinction by replacing non-canonical URLs with versions ^[15] .
• trying to automatically fix broken links.
• Creating permalinks stops the formation of broken links by ensuring that content is not migrated in the foreseeable future. Another type of creating permalinks is the permalink link, which then redirects to the actual content, which ensures that the link is saved even if the real content is moved to another place, so that links pointing to the resource remain unchanged.
• Design URLs - for example, semantic URLs - so that they do not need to be changed when another person begins to serve the document, or when other software is used on the server ^[16] .

User side

• The Linkgraph widget determines the URL of the correct page based on the old bit of the URL by using historical location information.
• The Google 404 Widget widget attempts to “guess” the correct URL and gives the user a dialog box to search for the correct page.
• When the user receives the code 404, the Google toolbar tries to help the user find the missing page ^[17] .

Website Archiving

To counter link extinction, website archiving is actively used to save web pages or parts of the network and to ensure that a set of pages is stored in archives , such as the , for future researchers, historians, and society. The goal of Internet archiving is to create an archive of the entire network by periodically taking snapshots of pages, which can then be freely accessed through the Wayback Machine . In January 2013, the company announced that it had reached a milestone of 240 billion archived URLs ^[18] . National libraries , and other organizations are also involved in archiving culturally important Web content.

Individual citizens can use many tools that allow them to archive web resources that may become unavailable in the future:

• The WayBack Machine, a not-for-profit organization Internet Archive ^[19] , is a free website that archives old web pages. It does not archive websites whose owners indicate that they do not want their website to be archived.
• A WebCite tool specifically designed for scientific authors, journal editors, and publishers for on-demand archiving and fetching links on the Internet ^[14] .
• Archive.is archive site saves snapshots of web pages. It retrieves one page per request, but, unlike WebCite, it includes Web 2.0 sites such as Google Maps and Twitter.
• The service, supported by Harvard Law School, together with a broad coalition of university libraries, takes a snapshot of the content URL and returns a permalink ^[1] .
• The Hiberlink project, created by the University of Edinburgh in collaboration with the Los Alamos National Laboratory and other organizations, works to measure the “extinction of links” in online scientific articles, as well as determine where the web content has been archived ^[20] . The related project, Memento, has set the technical standard for accessing online content as it existed in the past ^[21] .
• Some social bookmarking websites allow users to make an online clone of any webpage on the Internet by creating a copy with an independent url that remains accessible even if the original page has ceased to exist.
• The Amber tool, created at Harvard at the , is a tool to combat link extinction by archiving to WordPress and Drupal to prevent censorship of the network and support archiving ^[22] .

However, such conservation systems may experience turning off / on the service, so that the saved URLs periodically become unavailable ^[23] .

• Future-Proofing Your URIs
• To Nielsen Jakob, «Fighting Linkrot» , Jakob to Nielsen's Alertbox , June 14, 1998.