URI Transformation

Even though a crawler such as WebCopy may not support JavaScript, it may be possible to find pre-process URI's and transform them.

The example buttons below each have call a JavaScript function to navigate to a different page. Each link calls a JavaScript function openPage with two arguments. These arguments are then concatenated to create a destination URL which the browser is then instructed to navigate to. WebCopy can detect these patterns and generate the concatenated value, allowing it to crawl these pages, including correctly remapping the original source links to jump directly to the destination page.

Tests

Categtory 1 Index Categtory 1 Page 2 Categtory 2 Index Middleman Redirect

Learn More