Set stores the []byte representation of a response against a key

Get returns the []byte representation of a cached response and a bool set to true if the value isn't empty

Do sends an HTTP request

NewGeziyor creates new Geziyor with default values. If options provided, options

Start starts scraping

NewRequest returns a new Request given a method, URL, and optional body.

getFreshness will return one of fresh/stale/transparent based on the cache-control values of the request and the response fresh indicates the respons

Delete removes the value associated with the key

NewMemoryCacheTransport returns a new Transport using the in-memory cache implementation

DoRequest selects appropriate request handler, client or Chrome

SetDefaultHeader sets header if not exists before

newClientDefault creates new client with default options

Cancel request

ConvertMapToHeader converts map[string]interface{} to http.Header

DefaultString returns first non-empty string

NewClient creates http.Client with modified values for typical web scraper

RoundTrip is a wrapper for caching requests. If there is a fresh Response already in cache, then it will be returned without connecting to the server.

cacheKey returns the cache key for req.

CachedResponse returns the cached http.Response for req if present, and nil otherwise.

ConvertHeaderToMap converts http.Header to map[string]interface{}

Cookies returns the cookies to send in a request for the given URL.

Date parses and returns the value of the Date header.

New returns a new Cache that will store files in basePath

New returns a new Cache that will store items in an in-memory map

PleaseCache excercises a Cache implementation.

Post issues a POST to the specified URL

Read reads the next len(p) bytes from R or until R is drained. The return value n is the number of bytes read. If R has no data to return, err is io.E

cloneRequest returns a clone of the provided *http.Request. The clone is a shallow copy of the struct and its Header map. (This function copyright goa

Do sends an HTTP request

headerAllCommaSepValues returns all comma-separated values (each with whitespace trimmed) for header name in headers. According to Section 4.2 of the

Client returns an *http.Client that caches responses.

ContainsInt checks whether []int contains int

ContainsString checks whether []string ContainsString string

DefaultRune returns first non-empty rune

Export exports response data as CSV streaming file

Export exports response data as JSON streaming file

Export exports response data as JSON

Get issues a GET to the specified URL.

GetRendered issues GET request using headless browser Opens up a new Chrome instance, makes request, waits for rendering HTML DOM and closed. Rendered

Head issues a HEAD to the specified URL

IsHTML checks if response content is HTML by looking content-type header

JoinURL joins base response URL and provided relative URL. DEPRECATED: Use response.Request.URL.Parse(relativeURL) instead.

New returns a new Cache that will store leveldb in path

NewMetrics creates new metrics with given metrics.Type

NewRedirectionHandler returns maximum allowed redirection function with provided maxRedirect

NewTransport returns a new Transport with the provided Cache implementation and MarkCachedResponses set to true

RoundRobinProxy creates a proxy switcher function which rotates ProxyURLs on every request. The proxy type is determined by the URL scheme. "http", "h

RoundTripDummy has no awareness of any HTTP Cache-Control directives. Every request and its corresponding response are cached. When the same request i

RoundTripRFC2616 provides a RFC2616 compliant HTTP cache, i.e. with HTTP Cache-Control awareness, aimed at production and used in continuous runs to a

StartMetricsServer starts server that handles metrics Prometheus: http://localhost:2112/metrics Expvar : http://localhost:2112/debug/vars

Returns true if either the request or the response includes the stale-if-error cache control extension: https://tools.ietf.org/html/rfc5861

doRequestChrome opens up a new chrome instance and makes request

doRequestClient is a simple wrapper to read response according to options.

interruptSignalWaiter waits data from provided channels and stops scraper if shutdownChan channel receives SIGINT

jsonMarshalLine adds tab and comma around actual data

recoverMe prevents scraping being crashed. Logs error and stack trace

varyMatches will return false unless all of the cached values for the headers listed in Vary match the new request

Make sure to increase open file descriptor limits before running

Delete removes the response with key from the cache

Delete removes the response with key from the cache

Delete removes key from the cache

Export logs exported data to console as pretty printed

Get returns the response corresponding to key if present

Get returns the response corresponding to key if present

Get returns the []byte representation of the response and true if present, false if not

NewWithDB returns a new Cache using the provided leveldb as underlying storage.

NewWithDiskv returns a new Cache using the provided Diskv as underlying storage.

Set saves a response to the cache as key

Set saves a response to the cache as key