CrawlerRunConfig Parameters Documentation
Content Processing Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
word_count_threshold |
int | 200 | Minimum word count threshold before processing content |
extraction_strategy |
ExtractionStrategy | None | Strategy to extract structured data from crawled pages. When None, uses NoExtractionStrategy |
chunking_strategy |
ChunkingStrategy | RegexChunking() | Strategy to chunk content before extraction |
markdown_generator |
MarkdownGenerationStrategy | None | Strategy for generating markdown from extracted content |
content_filter |
RelevantContentFilter | None | Optional filter to prune irrelevant content |
only_text |
bool | False | If True, attempt to extract text-only content where applicable |
css_selector |
str | None | CSS selector to extract a specific portion of the page |
excluded_tags |
list[str] | [] | List of HTML tags to exclude from processing |
keep_data_attributes |
bool | False | If True, retain data-* attributes while removing unwanted attributes |
remove_forms |
bool | False | If True, remove all <form> elements from the HTML |
prettiify |
bool | False | If True, apply fast_format_html to produce prettified HTML output |
Caching Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
cache_mode |
CacheMode | None | Defines how caching is handled. Defaults to CacheMode.ENABLED internally |
session_id |
str | None | Optional session ID to persist browser context and page instance |
bypass_cache |
bool | False | Legacy parameter, if True acts like CacheMode.BYPASS |
disable_cache |
bool | False | Legacy parameter, if True acts like CacheMode.DISABLED |
no_cache_read |
bool | False | Legacy parameter, if True acts like CacheMode.WRITE_ONLY |
no_cache_write |
bool | False | Legacy parameter, if True acts like CacheMode.READ_ONLY |
Page Navigation and Timing Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
wait_until |
str | "domcontentloaded" | The condition to wait for when navigating |
page_timeout |
int | 60000 | Timeout in milliseconds for page operations like navigation |
wait_for |
str | None | CSS selector or JS condition to wait for before extracting content |
wait_for_images |
bool | True | If True, wait for images to load before extracting content |
delay_before_return_html |
float | 0.1 | Delay in seconds before retrieving final HTML |
mean_delay |
float | 0.1 | Mean base delay between requests when calling arun_many |
max_range |
float | 0.3 | Max random additional delay range for requests in arun_many |
semaphore_count |
int | 5 | Number of concurrent operations allowed |
Page Interaction Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
js_code |
str or list[str] | None | JavaScript code/snippets to run on the page |
js_only |
bool | False | If True, indicates subsequent calls are JS-driven updates |
ignore_body_visibility |
bool | True | If True, ignore whether the body is visible before proceeding |
scan_full_page |
bool | False | If True, scroll through the entire page to load all content |
scroll_delay |
float | 0.2 | Delay in seconds between scroll steps if scan_full_page is True |
process_iframes |
bool | False | If True, attempts to process and inline iframe content |
remove_overlay_elements |
bool | False | If True, remove overlays/popups before extracting HTML |
simulate_user |
bool | False | If True, simulate user interactions for anti-bot measures |
override_navigator |
bool | False | If True, overrides navigator properties for more human-like behavior |
magic |
bool | False | If True, attempts automatic handling of overlays/popups |
adjust_viewport_to_content |
bool | False | If True, adjust viewport according to page content dimensions |
Media Handling Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
screenshot |
bool | False | Whether to take a screenshot after crawling |
screenshot_wait_for |
float | None | Additional wait time before taking a screenshot |
screenshot_height_threshold |
int | 20000 | Threshold for page height to decide screenshot strategy |
pdf |
bool | False | Whether to generate a PDF of the page |
image_description_min_word_threshold |
int | 50 | Minimum words for image description extraction |
image_score_threshold |
int | 3 | Minimum score threshold for processing an image |
exclude_external_images |
bool | False | If True, exclude all external images from processing |
Link and Domain Handling Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
exclude_social_media_domains |
list[str] | SOCIAL_MEDIA_DOMAINS | List of domains to exclude for social media links |
exclude_external_links |
bool | False | If True, exclude all external links from the results |
exclude_social_media_links |
bool | False | If True, exclude links pointing to social media domains |
exclude_domains |
list[str] | [] | List of specific domains to exclude from results |
Debugging and Logging Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
verbose |
bool | True | Enable verbose logging |
log_console |
bool | False | If True, log console messages from the page |