# Web Crawling

**Overview**

The **web crawling** **feature** in FormWise allows users to scrape text content from a specific URL and use that information within their **Copilots** or **SmartForms**. This feature is available **only for GPT-4 models and above**, ensuring the most powerful processing capabilities for text analysis and interaction.

#### How the Web Crawling Feature Works

When you input a URL into your FormWise tool, the web crawling feature scrapes all the text-based content on that webpage. It does not extract images, videos, or other non-text elements but focuses on gathering readable text content for analysis or integration into prompts.

The way this works depends on the tool you're using:

1. **In Copilots:**

* The web crawling is activated when the user provides a specific URL. The Copilot will then scrape all text from that URL and use it in real-time to generate responses, suggestions, or actions based on the content of the webpage.

2. **In SmartForms:**

* Web crawling happens when the **prompt instructions** include a URL. This can be done in two ways:

-Mapping the URL as one of the input fields for the form.

-Adding the URL directly in the prompt instructions to guide the SmartForm on where to extract text.

#### Important Notes:

* This feature is **only available for GPT-4 models** and higher, ensuring that the scraped content is processed with maximum efficiency and accuracy.
* Web scraping is particularly useful for tools that need real-time, external data from sources like blogs, articles, or product pages to generate dynamic responses or recommendations.

#### Scenario Example

Let’s say you’re using a Copilot to help manage blog content creation. You want the **Copilot** to summarize key points from an article on SEO trends to help you generate ideas for a new blog post. When a URL is given in conversation to the Copilot, and the web crawler scrapes the entire text of that page. The Copilot then generates a summary and suggests relevant blog titles, using the scraped content as a source.

Similarly, in a **SmartForm**, if you’re creating a form that helps users compare products, you can include an input field where the user adds a product page URL. The SmartForm will scrape all relevant product information from the URL and use that data to fill in the comparison details.

This feature makes your FormWise tools incredibly versatile, allowing them to interact with real-time data from across the web.

By leveraging the power of GPT-4, you can ensure that your Copilot or SmartForm processes and outputs this information effectively.

<br>


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://manual.formwise.ai/formwise/advanced-features/web-intelligence/web-crawling.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.