Package org.apache.manifoldcf.crawler.connectors.webcrawler

Interface IThrottledConnection

    • Method Detail

      • setAbortChecker

        void setAbortChecker​(AbortChecker abortCheck)
        Set the abort checker. This must be done before the connection is actually used.

      • hasExpired

        boolean hasExpired​(long currentTime)
        Check whether the connection has expired.
        Parameters:
        currentTime - is the current time to use to judge if a connection has expired.
        Returns:
        true if the connection has expired, and should be closed.

      • beginFetch

        void beginFetch​(java.lang.String fetchType)
         throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                org.apache.manifoldcf.agents.interfaces.ServiceInterruption
        Begin the fetch process.
        Parameters:
        fetchType - is a short descriptive string describing the kind of fetch being requested. This is used solely for logging purposes.
        Throws:
        org.apache.manifoldcf.core.interfaces.ManifoldCFException
        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

      • executeFetch

        void executeFetch​(java.lang.String urlPath,
                  java.lang.String userAgent,
                  java.lang.String from,
                  boolean redirectOK,
                  java.lang.String host,
                  FormData formData,
                  LoginCookies loginCookies)
           throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                  org.apache.manifoldcf.agents.interfaces.ServiceInterruption
        Execute the fetch and get the return code. This method uses the standard logging mechanism to keep track of the fetch attempt. It also signals the following conditions: ServiceInterruption (if a dynamic error occurs), or ManifoldCFException if a fatal error occurs, or nothing if a standard protocol error occurs. Note that, for proxies etc, the idea is for this fetch request to handle whatever redirections are needed to support proxies.
        Parameters:
        urlPath - is the path part of the url, e.g. "/robots.txt"
        userAgent - is the value of the userAgent header to use.
        from - is the value of the from header to use.
        redirectOK - should be set to true if you want redirects to be automatically followed.
        host - is the value to use as the "Host" header, or null to use the default.
        formData - describes additional form arguments and how to fetch the page.
        loginCookies - describes the cookies that should be in effect for this page fetch.
        Throws:
        org.apache.manifoldcf.core.interfaces.ManifoldCFException
        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

      • getResponseCode

        int getResponseCode()
             throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                    org.apache.manifoldcf.agents.interfaces.ServiceInterruption
        Get the http response code.
        Returns:
        the response code. This is either an HTTP response code, or one of the codes above.
        Throws:
        org.apache.manifoldcf.core.interfaces.ManifoldCFException
        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

      • getLastFetchCookies

        LoginCookies getLastFetchCookies()
                          throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                 org.apache.manifoldcf.agents.interfaces.ServiceInterruption
        Get the last fetch cookies.
        Returns:
        the cookies now in effect from the last fetch.
        Throws:
        org.apache.manifoldcf.core.interfaces.ManifoldCFException
        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

      • getResponseHeaders

        java.util.Map<java.lang.String,​java.util.List<java.lang.String>> getResponseHeaders()
                                                                                   throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                                                                          org.apache.manifoldcf.agents.interfaces.ServiceInterruption
        Get response headers
        Returns:
        a map keyed by header name containing a list of values.
        Throws:
        org.apache.manifoldcf.core.interfaces.ManifoldCFException
        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

      • getResponseHeader

        java.lang.String getResponseHeader​(java.lang.String headerName)
                            throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                   org.apache.manifoldcf.agents.interfaces.ServiceInterruption
        Get a specified response header, if it exists.
        Parameters:
        headerName - is the name of the header.
        Returns:
        the header value, or null if it doesn't exist.
        Throws:
        org.apache.manifoldcf.core.interfaces.ManifoldCFException
        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

      • getResponseBodyStream

        java.io.InputStream getResponseBodyStream()
                                   throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                          org.apache.manifoldcf.agents.interfaces.ServiceInterruption
        Get the response input stream. It is the responsibility of the caller to close this stream when done.
        Throws:
        org.apache.manifoldcf.core.interfaces.ManifoldCFException
        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

      • getLimitedResponseBody

        java.lang.String getLimitedResponseBody​(int maxSize,
                                        java.lang.String encoding)
                                 throws org.apache.manifoldcf.core.interfaces.ManifoldCFException,
                                        org.apache.manifoldcf.agents.interfaces.ServiceInterruption
        Get limited response as a string.
        Throws:
        org.apache.manifoldcf.core.interfaces.ManifoldCFException
        org.apache.manifoldcf.agents.interfaces.ServiceInterruption

      • noteInterrupted

        void noteInterrupted​(java.lang.Throwable e)
        Note that the connection fetch was interrupted by something.

      • doneFetch

        void doneFetch​(org.apache.manifoldcf.crawler.interfaces.IProcessActivity activities)
        throws org.apache.manifoldcf.core.interfaces.ManifoldCFException
        Done with the fetch. Call this when the fetch has been completed. A log entry will be generated describing what was done.
        Throws:
        org.apache.manifoldcf.core.interfaces.ManifoldCFException

      • close

        void close()
        Close the connection. Call this to return the connection to its pool.

      • destroy

        void destroy()
        Destroy the connection. Call this to close the connection.