edu.uci.ics.crawler4j.robotstxt

Class RobotstxtServer

java.lang.Object

edu.uci.ics.crawler4j.robotstxt.RobotstxtServer

public class RobotstxtServer
extends Object

Author:

Yasser Ganjisaffar

Field Summary

Fields

Modifier and Type	Field and Description
protected RobotstxtConfig	config
protected Map<String,HostDirectives>	host2directivesCache
protected PageFetcher	pageFetcher

Constructor Summary

Constructors

Constructor and Description
RobotstxtServer(RobotstxtConfig config, PageFetcher pageFetcher)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	allows(WebURL webURL)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

protected RobotstxtConfig config

host2directivesCache

protected final Map<String,HostDirectives> host2directivesCache

pageFetcher

protected PageFetcher pageFetcher

Constructor Detail

RobotstxtServer

public RobotstxtServer(RobotstxtConfig config,
               PageFetcher pageFetcher)

Method Detail

allows

public boolean allows(WebURL webURL)