Meta elements are tags used in HTML and XHTML documents to provide structured metadata about a Web page. They are part of a web page's head section. Multiple Meta elements with different attributes can be used on the same page. Meta elements can be used to specify page description, keywords and any other metadata not provided through the other head elements and attributes.
The meta element has two uses: either to emulate the use of an HTTP response header field, or to embed additional metadata within the HTML document. It is often the former usage that has particular interest to a web crawler.
HTTP Crawler Test Site