Package org.apache.manifoldcf.crawler.connectors.webcrawler

Class RobotsManager.RobotsData

  • Enclosing class:
    RobotsManager
    protected static class RobotsManager.RobotsData
extends java.lang.Object
    This is a cached data item.

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected long expiration  
      protected java.util.ArrayList records  

    • Constructor Summary

      Constructors 
      Constructor Description
      RobotsData​(java.io.InputStream is, long expiration, java.lang.String hostName, org.apache.manifoldcf.crawler.interfaces.IProcessActivity activities)
      Constructor.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      long getExpirationTime()
      Get expiration
      boolean isFetchAllowed​(java.lang.String userAgent, java.lang.String pathString)
      Check if fetch is allowed
      protected void parseRobotsTxt​(java.io.BufferedReader r, java.lang.String hostName, org.apache.manifoldcf.crawler.interfaces.IProcessActivity activities)
      Parse the robots.txt file using a reader.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • expiration

        protected long expiration

      • records

        protected java.util.ArrayList records

    • Constructor Detail

      • RobotsData

        public RobotsData​(java.io.InputStream is,
                  long expiration,
                  java.lang.String hostName,
                  org.apache.manifoldcf.crawler.interfaces.IProcessActivity activities)
           throws java.io.IOException,
                  org.apache.manifoldcf.core.interfaces.ManifoldCFException
        Constructor.
        Throws:
        java.io.IOException
        org.apache.manifoldcf.core.interfaces.ManifoldCFException

    • Method Detail

      • isFetchAllowed

        public boolean isFetchAllowed​(java.lang.String userAgent,
                              java.lang.String pathString)
        Check if fetch is allowed

      • getExpirationTime

        public long getExpirationTime()
        Get expiration

      • parseRobotsTxt

        protected void parseRobotsTxt​(java.io.BufferedReader r,
                              java.lang.String hostName,
                              org.apache.manifoldcf.crawler.interfaces.IProcessActivity activities)
                       throws java.io.IOException,
                              org.apache.manifoldcf.core.interfaces.ManifoldCFException
        Parse the robots.txt file using a reader. Is NOT expected to close the stream.
        Throws:
        java.io.IOException
        org.apache.manifoldcf.core.interfaces.ManifoldCFException