Jurisdiction(s):
United Kingdom

Related legal acts

Key definition

Index definition

What does Index mean? In legal practice, an index is a publicly available numerical measure tracking the value or performance of specified underlying assets, prices or data, used as a reference point in contracts, securities documentation and regulation. In financial services, the term is defined in the UK Benchmarks Regulation and the EU Benchmarks Regulation; when an index is used to determine amounts payable under a financial instrument or contract, or to measure a fund’s performance, it is treated as a benchmark, engaging regulatory and licensing requirements. Market indices (for example, the FTSE All-Share Index, which tracks the performance of most companies listed on...

Web crawling, indexing, caching and scraping: UK legal issues for website operators—copyright, database right, website terms, data protection, robots.txt, Computer Misuse Act, and post-Brexit/EU considerations

Published by a LexisNexis TMT expert

Practice notes

$imgtext$

This Practice Note sets out how intellectual property and related rights, together with data protection duties, operate in the setting of web crawling, indexing, caching and scraping, viewed from a website operator’s standpoint.

An explanation of the terminology

The principal concepts of crawling, indexing, caching and scraping are outlined below.

Crawling

Web crawling is the activity whereby automated programmes (often called ‘bots’, ‘spiders’ or simply ‘web crawlers’) are deployed to traverse and read information across the web.
One well-known crawler is Googlebot, which Google uses to copy web pages onto its servers; Google then indexes them (see the section on Indexing) to support searching of the internet. Website operators often provide ‘sitemaps’ (an XML file listing all pages on a site) to assist and enhance search engines’ crawling.
Crawlers can also be put to other uses such as ‘scraping’ (see the section on ‘Scraping’) or collecting email addresses to send unsolicited emails (i.e. spamming).
In addition to search engines, web crawling is also utilised by meta-sites,...

To view the latest version of this document and thousands of others like it, sign-in with LexisNexis or register for a free trial.

Toby Headdon

Bristows

Toby advises on all aspects of intellectual property including dispute resolution, sale and acquisition, licensing and general advisory work. He has acted for clients across a range of sectors including publishing, technology, finance, utilities, real estate and the public sector. He also represents clients in IT disputes.Toby is recommended by both Chambers & Partners and the Legal 500 directories for intellectual property. He is a professional member of The Law Society, the Society for Computers and Law, the Competition Law Association and the British Literary and Artistic Copyright Association. He also holds Master’s degrees in law from Columbia University (New York), where he was a Fulbright Scholar, and the University of Oxford....

Hannah Crowther

Bristows

Hannah Crowther's practice involves advising on a wide variety of data protection issues, ranging from enterprise-wide General Data Protection Regulation compliance projects and Binding Corporate Rules, to responding to individual subject access requests or complaints. Hannah's practice includes a significant amount of contentious work, advising clients on managing data breaches and investigations by the UK Information Commissioner and other EU Data Protection Authorities. She is also experienced in data protection litigation, assisting with cases before the UK Court of Appeal and the Court of Justice of the European Union. Hannah regularly advises her clients in relation to their marketing initiatives and compliance with the UK and EU rules on direct marketing. ...

Web page updated on 21/05/2026

Popular documents

Using the Judicial College Guidelines to assess general damages in personal injury claims

When evaluating a general damages claim, the practitioner ought initially to refer to the Judicial College Guidelines (JCG)...

Tomlin orders in England and Wales: CPR guidance on drafting, confidentiality, approval, variation, enforcement, limitation, CCA issues, court-specific requirements, sample wording and precedents

This Practice Note This Practice Note reviews mechanisms used in settling litigation. A Tomlin order consists of a consent order paired with a schedule. It operates to stay proceedings on terms that have been agreed. The provisions contained in the schedule may remain confidential. This Practice Note describes the scope of confidentiality attaching to the schedule and sets out how it differs from a standard consent order. Sample wording for a Tomlin order is included, alongside links to precedents, as well as guidance on court approval. It also addresses varying, setting aside and enforcing a Tomlin order, including the considerations the court will take into account when handling applications for each. Further guidance is provided on interpreting and applying the relevant provisions of the CPR; however, some courts and divisions impose very specific requirements for both drafting and approval, and for approaching the schedule and confidentiality issues. Accordingly, you must consider the particular rules and court guide provisions in the forum where your claim is proceeding when drawing up the Tomlin order...

Precedent: Deed of Surrender of Lease with Optional Premium, Settlement and Release, Landlord, Mortgagee and Guarantor Consents, Reimbursement and HM Land Registry Provisions (England and Wales)

Date [ date ] Parties [ name of Landlord ] [ of OR incorporated in England and Wales (company registration number [ number ]) with its registered office at ] [ address ] (Landlord) [ name of Tenant ] [ of OR incorporated in England and Wales (company registration number [ number ]) with its registered office at ] [ address ] (Tenant) [ [ name of Guarantor ] [ of OR incorporated in England and Wales (company registration number [ number ]) with its registered office at ] [ address ] (Guarantor) ] [ [ name of Mortgagee ] [ of OR incorporated in England and Wales (company registration number [ number ]) with its registered office at ] [ address ] (Mortgagee) ] Definitions Within this Deed, the terms below shall be interpreted as follows: [ Annual Rent • the annual sum reserved under the Lease; ] [ Insurance Rent • the Tenant’s share of the Landlord’s costs of insuring the Property (as set out in the Lease); ] Lease • the lease of the Property dated [ date ], entered into between (1) [ the Landlord OR [ name ...

Precedent Statutory Declaration: Standard Wording under the Statutory Declarations Act 1835

I, [ name ], of [ address ], solemnly and sincerely state that: [ Matters to be verified, set out in numbered paragraphs ] I make this solemn statement in good conscience, believing it to be true, and pursuant to the provisions of the Statutory Declarations Act 1835. DECLARED at [ details ] this [ day ] day of [ month and year ] Before me ................................................................................ [ signature of the person before whom the declaration is made ] A [ commissioner for oaths OR [ solicitor OR [ insert other qualification ] ] authorised to administer oaths ]...