HTTP Crawler Test Site

This website was created to replace the original static website used for testing WebCopy features, expanded to cover additional scenarios that the original didn't handle and to ensure a lean site that doesn't slow down integration testing unnecessarily. Of course, writing better tests is just as important! But every little helps, over time it has been used to find bugs in WebCopy's crawl engine and to help ensure regressions don't make it back in.

Cyotek is a .NET shop and usually all our products are written in C# using Visual Studio. This time however we decided to keep it simple and this site has been written with PHP, originally using WebMatrix and currently Visual Studio Code. As we aren't PHP coders and don't have much of a clue on best practices for writing PHP or even most of the language the code quality is probably less than stellar. Although it was fairly fun to write as it doesn't need an ever increasing amount of binary dependencies, no assembly bindings or version woes... it just works.

You can view a live demo running at https://demo.cyotek.com. To run the source you should only need PHP 5.3 or above and some form of web server. This site has been tested using IIS Express, IIS 8.5, IIS 10 and Apache running on some variant of Linux.


CSS has been taken from Primer by GitHub.

Various bits of PHP were lifted from the PHP user manual.

The fav icon is derived from Material Core Check.

Markdown processing is handled by PHP Markdown Lib.

Some text has been borrowed from Wikipedia and the Mozilla Developer Network.

The srcset demonstration was derived from a WebKit.org demo.

The skyline images were generated by the imaginatively titled Skyline Generator.

The photograph of St Paul's Cathedral in London taken by Vadim Sherbakov.

Generic Avatar graphic by WingTillDie

Formatting of the raw code files is being gradually performed via VSCodeBeautify

Source Code

The source code for this site is available from Cyotek's GitHub page.


This site and its source code is licensed via the MIT License. View License.