{"info":{"_postman_id":"6d5c07cc-d0c4-4927-bff1-c70112598f28","name":"Link Checker","description":"

This collection will check all the links on a webpage, crawl all internal links, and report broken links (if any).

\n

Click the Run in Postman button to import the sample collection and environment template into your Postman app. You should now see the collection in the sidebar to the left and the environment selected in the dropdown in the top right.

\n

\"import

\n

Get Started:

\n
    \n

  1. Update environment: Click the Quick Look icon in the top right to view and edit the environment variables. This is where you can update the values to check links on your own website. In many cases, your root_url will be the same as your start_url. However, in this example, we will use https://learning.getpostman.com as our root_url, and start checking links on https://learning.getpostman.com/docs/postman/launching_postman/installation_and_updates.

    \n

    \"environment

    \n
    2. \n

  2. Open Postman Console: This step is optional. If you want to see a stream of requests and view any logged statements, go to the application menu, and select View > Show Postman Console to open the console in a separate window. Do this before you send any requests or run the collection.

    \n
    3. \n
\n

\"Postman

\n
    \n
  1. Run collection: Click the right angle bracket (>) to expand the collection details view. Click the Run button to open the collection runner in a separate window.
    2. \n
\n

\"collection

\n

Verify that your collection and environment are selected in the respective dropdowns, and click Start Run to begin running your collection.

\n

\"collection

\n

You should now see your tests running and passing, crawling all the links until there are no more links to check.

\n

\"tests

\n","schema":"https://schema.getpostman.com/json/collection/v2.0.0/collection.json","toc":[{"content":"Get Started:","slug":"get-started"}],"owner":"3967924","collectionId":"6d5c07cc-d0c4-4927-bff1-c70112598f28","publishedId":"RW1bozae","public":true,"customColor":{"top-bar":"FFFFFF","right-sidebar":"303030","highlight":"EF5B25"},"publishDate":"2018-05-01T14:06:48.000Z"},"item":[{"name":"Initialize","event":[{"listen":"prerequest","script":{"type":"text/javascript","exec":["// Custom pre-request scripts can be written in JavaScript","","// Before running the collection, check for required environment variables","// Throw an error if any are missing","const postmanEnvironment = pm.environment.toObject();","","// Make sure required environment keys are present","pm.expect(postmanEnvironment).to.have.all.keys(\"startUrl\", \"rootUrl\");","","// Make sure environment values look like URLs","const urlRegex = /^https?:\\/\\//;","pm.expect(postmanEnvironment.startUrl, 'startUrl does not match URL pattern').to.match(urlRegex);","pm.expect(postmanEnvironment.rootUrl, 'rootUrl does not match URL pattern').to.match(urlRegex);","","// Set global variables to default values","// BEST PRACTICE: Use global variables for temporary state in your collections that needs","// to be shared between requests. Use environment variables for configuration values.","pm.globals.set(\"links\", \"[]\");","pm.globals.set(\"url\", postmanEnvironment.startUrl);","pm.globals.set(\"index\", -1);"]}}],"id":"098e94a2-29b2-4cf5-8bf5-8a445b9e8bd5","request":{"method":"GET","header":[],"body":{"mode":"raw","raw":""},"url":"{{startUrl}}","description":"

Under the Tests tab, we will set a few more environment variables that will be used in the subsequent request, using the setEnvironmentVariable() method.

\n

\"setEnvironmentVariable\"

\n","urlObject":{"host":["{{startUrl}}"],"query":[],"variable":[]}},"response":[],"_postman_id":"098e94a2-29b2-4cf5-8bf5-8a445b9e8bd5"},{"name":"Check URL","event":[{"listen":"test","script":{"type":"text/javascript","exec":["// Tests and custom scripts can be written in JavaScript","","// Get environment variables","const startUrl = pm.environment.get(\"startUrl\");","const rootUrl = pm.environment.get(\"rootUrl\");","","// Get global variables","// BEST PRACTICE: Use global variables for temporary state in your collections that needs","// to be shared between requests. Use environment variables for configuration values.","const links = JSON.parse(pm.globals.get(\"links\"));","const currentUrl = pm.globals.get(\"url\");","const currentIndex = parseInt(pm.globals.get(\"index\"));","","// Test if a link works","pm.test(`Link to \"${currentUrl}\" works`, function () {"," try {"," // Check that the response code is a non-error code"," pm.response.to.not.be.error;"," }"," catch (error) {"," // We got an error response, so log more details to help with debugging"," console.log(`FAILED :: ${currentUrl}`);"," console.log(`FAILED :: status code is ${pm.response.code}`);"," "," // Rethrow the error to fail the test"," throw error;"," }","});","","if (currentUrl.includes(startUrl)) {"," // This is an internal link and we should crawl it for more links"," // Load the response body as HTML using cheerio"," const $ = cheerio.load(pm.response.text());"," "," // Get the links from all the tags"," $(\"a\").each(function () {"," const newLink = $(this).attr(\"href\");"," "," // Add links to the links array if not already in there"," // NOTE: Links only need to be checked once, so repeated links are excluded. If you wanted to exclude"," // some other link types (for example, ads), you could add the appropriate exclusion criteria here."," if (!links.includes(newLink)) {"," links.push(newLink);"," }"," });","}","","// Check the next link url","const [nextUrl, nextIndex] = getNextUrlAndIndex(links, currentIndex);","","if (nextUrl) {"," // Update global variables before the next iteration"," pm.globals.set(\"links\", JSON.stringify(links));"," pm.globals.set(\"url\", nextUrl);"," pm.globals.set(\"index\", nextIndex);"," "," // Continue calling the same request until all links are checked"," // CAREFUL! This causes the request to loop until there is no nextUrl"," postman.setNextRequest(\"Check URL\");","}","else {"," // We've gone through all the links"," console.log(\"No more links to check!\");"," "," // Clear global variables used by link checker"," pm.globals.clear(\"links\");"," pm.globals.clear(\"url\");"," pm.globals.clear(\"index\");"," "," // End of loop"," postman.setNextRequest(null);","}","","/**"," * Get the indexed link from an array of links and handle relative links."," */","function getNextUrlAndIndex (links = [], index = 0) {"," const nextIndex = index + 1;"," "," if (links.length - 1 === nextIndex) {"," // There is no next link"," return [];"," }"," "," const linkUrl = links[nextIndex];"," "," // NOTE: You can use a similar check to skip other categories of links,"," // such as download links or external links."," if (!linkUrl) {"," // Skip null links"," console.log('WARNING :: encountered a null link.');"," "," // Recursively try to get the next link"," return getNextUrlAndIndex(links, nextIndex);"," }"," "," if (/^https?:\\/\\//.test(linkUrl)) {"," // This isn't a relative link, so return it"," return [linkUrl, nextIndex];"," }"," "," // If the link is relative, prepend with rootUrl"," const cleanedRoot = rootUrl.replace(/\\/$/, '');"," const cleanedPath = linkUrl.replace(/^[\\.\\/]+/, '');"," "," return [[cleanedRoot, cleanedPath].join('/'), nextIndex];","}",""]}}],"id":"82f93e64-3b79-4564-87a0-98c83c024fca","request":{"method":"GET","header":[],"body":{"mode":"raw","raw":""},"url":"{{url}}","description":"

This request will crawl internal links and check all links for a healthy HTTP status code that is less than 400.

\n

HTML scraping

\n

Finding all the links on a page requires scraping HTML. The Postman Sandbox supports cheerio as a library for scraping HTML elements. Read more about using the Postman Sandbox and other libraries and utilities supported in the pre-request and scripts sections.

\n

\"Cheerio

\n

Building workflows

\n

The setNextRequest() method accepts a request name or id within the same collection as a parameter. Use this method to establish a workflow sequence and designate which request in the same collection to run next, instead of defaulting to the linear execution. Read more about building workflows.

\n

In this example, we will continue calling the same request until all links are checked.

\n

\"setNextRequest

\n","urlObject":{"host":["{{url}}"],"query":[],"variable":[]}},"response":[],"_postman_id":"82f93e64-3b79-4564-87a0-98c83c024fca"}]}