CrawlConfig (crawler4j 3.5 API)

java.lang.Object
- edu.uci.ics.crawler4j.crawler.CrawlConfig

```
public class CrawlConfig
extends Object
```

Constructor Summary

Constructors
Constructor and Description

CrawlConfig()

Constructors
Constructor and Description
`CrawlConfig()`

Method Summary

Methods
Modifier and Type	Method and Description
`int`	`getConnectionTimeout()`
`String`	`getCrawlStorageFolder()`
`int`	`getMaxConnectionsPerHost()`
`int`	`getMaxDepthOfCrawling()`
`int`	`getMaxDownloadSize()`
`int`	`getMaxOutgoingLinksToFollow()`
`int`	`getMaxPagesToFetch()`
`int`	`getMaxTotalConnections()`
`int`	`getPolitenessDelay()`
`String`	`getProxyHost()`
`String`	`getProxyPassword()`
`int`	`getProxyPort()`
`String`	`getProxyUsername()`
`int`	`getSocketTimeout()`
`String`	`getUserAgentString()`
`boolean`	`isFollowRedirects()`
`boolean`	`isIncludeBinaryContentInCrawling()`
`boolean`	`isIncludeHttpsPages()`
`boolean`	`isResumableCrawling()`
`void`	`setConnectionTimeout(int connectionTimeout)` Connection timeout in milliseconds
`void`	`setCrawlStorageFolder(String crawlStorageFolder)` The folder which will be used by crawler for storing the intermediate crawl data.
`void`	`setFollowRedirects(boolean followRedirects)` Should we follow redirects?
`void`	`setIncludeBinaryContentInCrawling(boolean includeBinaryContentInCrawling)` Should we fetch binary content such as images, audio, ...?
`void`	`setIncludeHttpsPages(boolean includeHttpsPages)` Should we also crawl https pages?
`void`	`setMaxConnectionsPerHost(int maxConnectionsPerHost)` Maximum Connections per host
`void`	`setMaxDepthOfCrawling(int maxDepthOfCrawling)` Maximum depth of crawling For unlimited depth this parameter should be set to -1
`void`	`setMaxDownloadSize(int maxDownloadSize)` Max allowed size of a page.
`void`	`setMaxOutgoingLinksToFollow(int maxOutgoingLinksToFollow)` Max number of outgoing links which are processed from a page
`void`	`setMaxPagesToFetch(int maxPagesToFetch)` Maximum number of pages to fetch For unlimited number of pages, this parameter should be set to -1
`void`	`setMaxTotalConnections(int maxTotalConnections)` Maximum total connections
`void`	`setPolitenessDelay(int politenessDelay)` Politeness delay in milliseconds (delay between sending two requests to the same host).
`void`	`setProxyHost(String proxyHost)` If crawler should run behind a proxy, this parameter can be used for specifying the proxy host.
`void`	`setProxyPassword(String proxyPassword)` If crawler should run behind a proxy and user/pass is needed for authentication in proxy, this parameter can be used for specifying the password.
`void`	`setProxyPort(int proxyPort)` If crawler should run behind a proxy, this parameter can be used for specifying the proxy port.
`void`	`setProxyUsername(String proxyUsername)` If crawler should run behind a proxy and user/pass is needed for authentication in proxy, this parameter can be used for specifying the username.
`void`	`setResumableCrawling(boolean resumableCrawling)` If this feature is enabled, you would be able to resume a previously stopped/crashed crawl.
`void`	`setSocketTimeout(int socketTimeout)` Socket timeout in milliseconds
`void`	`setUserAgentString(String userAgentString)` user-agent string that is used for representing your crawler to web servers.
`String`	`toString()`
`void`	`validate()` Validates the configs specified by this instance.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Constructor Detail
  - CrawlConfig
```
public CrawlConfig()
```
- Method Detail
  - validate
```
public void validate()
              throws Exception
```
    Validates the configs specified by this instance.
    
    Throws:
    
    Exception
  - getCrawlStorageFolder
```
public String getCrawlStorageFolder()
```
  - setCrawlStorageFolder
```
public void setCrawlStorageFolder(String crawlStorageFolder)
```
    The folder which will be used by crawler for storing the intermediate crawl data. The content of this folder should not be modified manually.
  - isResumableCrawling
```
public boolean isResumableCrawling()
```
  - setResumableCrawling
```
public void setResumableCrawling(boolean resumableCrawling)
```
    If this feature is enabled, you would be able to resume a previously stopped/crashed crawl. However, it makes crawling slightly slower
  - getMaxDepthOfCrawling
```
public int getMaxDepthOfCrawling()
```
  - setMaxDepthOfCrawling
```
public void setMaxDepthOfCrawling(int maxDepthOfCrawling)
```
    Maximum depth of crawling For unlimited depth this parameter should be set to -1
  - getMaxPagesToFetch
```
public int getMaxPagesToFetch()
```
  - setMaxPagesToFetch
```
public void setMaxPagesToFetch(int maxPagesToFetch)
```
    Maximum number of pages to fetch For unlimited number of pages, this parameter should be set to -1
  - getUserAgentString
```
public String getUserAgentString()
```
  - setUserAgentString
```
public void setUserAgentString(String userAgentString)
```
    user-agent string that is used for representing your crawler to web servers. See http://en.wikipedia.org/wiki/User_agent for more details
  - getPolitenessDelay
```
public int getPolitenessDelay()
```
  - setPolitenessDelay
```
public void setPolitenessDelay(int politenessDelay)
```
    Politeness delay in milliseconds (delay between sending two requests to the same host).
    
    Parameters:
    politenessDelay - the delay in milliseconds.
  - isIncludeHttpsPages
```
public boolean isIncludeHttpsPages()
```
  - setIncludeHttpsPages
```
public void setIncludeHttpsPages(boolean includeHttpsPages)
```
    Should we also crawl https pages?
  - isIncludeBinaryContentInCrawling
```
public boolean isIncludeBinaryContentInCrawling()
```
  - setIncludeBinaryContentInCrawling
```
public void setIncludeBinaryContentInCrawling(boolean includeBinaryContentInCrawling)
```
    Should we fetch binary content such as images, audio, ...?
  - getMaxConnectionsPerHost
```
public int getMaxConnectionsPerHost()
```
  - setMaxConnectionsPerHost
```
public void setMaxConnectionsPerHost(int maxConnectionsPerHost)
```
    Maximum Connections per host
  - getMaxTotalConnections
```
public int getMaxTotalConnections()
```
  - setMaxTotalConnections
```
public void setMaxTotalConnections(int maxTotalConnections)
```
    Maximum total connections
  - getSocketTimeout
```
public int getSocketTimeout()
```
  - setSocketTimeout
```
public void setSocketTimeout(int socketTimeout)
```
    Socket timeout in milliseconds
  - getConnectionTimeout
```
public int getConnectionTimeout()
```
  - setConnectionTimeout
```
public void setConnectionTimeout(int connectionTimeout)
```
    Connection timeout in milliseconds
  - getMaxOutgoingLinksToFollow
```
public int getMaxOutgoingLinksToFollow()
```
  - setMaxOutgoingLinksToFollow
```
public void setMaxOutgoingLinksToFollow(int maxOutgoingLinksToFollow)
```
    Max number of outgoing links which are processed from a page
  - getMaxDownloadSize
```
public int getMaxDownloadSize()
```
  - setMaxDownloadSize
```
public void setMaxDownloadSize(int maxDownloadSize)
```
    Max allowed size of a page. Pages larger than this size will not be fetched.
  - isFollowRedirects
```
public boolean isFollowRedirects()
```
  - setFollowRedirects
```
public void setFollowRedirects(boolean followRedirects)
```
    Should we follow redirects?
  - getProxyHost
```
public String getProxyHost()
```
  - setProxyHost
```
public void setProxyHost(String proxyHost)
```
    If crawler should run behind a proxy, this parameter can be used for specifying the proxy host.
  - getProxyPort
```
public int getProxyPort()
```
  - setProxyPort
```
public void setProxyPort(int proxyPort)
```
    If crawler should run behind a proxy, this parameter can be used for specifying the proxy port.
  - getProxyUsername
```
public String getProxyUsername()
```
  - setProxyUsername
```
public void setProxyUsername(String proxyUsername)
```
    If crawler should run behind a proxy and user/pass is needed for authentication in proxy, this parameter can be used for specifying the username.
  - getProxyPassword
```
public String getProxyPassword()
```
  - setProxyPassword
```
public void setProxyPassword(String proxyPassword)
```
    If crawler should run behind a proxy and user/pass is needed for authentication in proxy, this parameter can be used for specifying the password.
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object

Class CrawlConfig

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

CrawlConfig

Method Detail

validate

getCrawlStorageFolder

setCrawlStorageFolder

isResumableCrawling

setResumableCrawling

getMaxDepthOfCrawling

setMaxDepthOfCrawling

getMaxPagesToFetch

setMaxPagesToFetch

getUserAgentString

setUserAgentString

getPolitenessDelay

setPolitenessDelay

isIncludeHttpsPages

setIncludeHttpsPages

isIncludeBinaryContentInCrawling

setIncludeBinaryContentInCrawling

getMaxConnectionsPerHost

setMaxConnectionsPerHost

getMaxTotalConnections

setMaxTotalConnections

getSocketTimeout

setSocketTimeout

getConnectionTimeout

setConnectionTimeout

getMaxOutgoingLinksToFollow

setMaxOutgoingLinksToFollow

getMaxDownloadSize

setMaxDownloadSize

isFollowRedirects

setFollowRedirects

getProxyHost

setProxyHost

getProxyPort

setProxyPort

getProxyUsername

setProxyUsername

getProxyPassword

setProxyPassword

toString