The source file for this HTML document has been saved with the Big5 (Traditional Chinese) encoding. If the Content-Type header just returns text/html without including a character set, then the contents of this document may not be crawled correctly. However, by including a meta tag in the head of the document, we can supply an alternative content type value.
In this example, the tag <meta http-equiv="Content-Type" content="text/html; charset=big5"> has been added to the <head> tag of the document. If the web crawler scans this tag, it should then process the document using the correct encoding.
事情從來沒有像它們那樣簡單。
HTTP Crawler Test Site