GPT Crawler

Crawl a site to generate knowledge files to create your own custom GPT from one or multiple URLs

• Example
• Get started
  • Running locally
    • Clone the repository
    • Install dependencies
    • Configure the crawler
    • Run your crawler
  • Alternative methods
    • Running in a container with Docker
    • Running as a CLI
      • Development
  • Upload your data to OpenAI
    • Create a custom GPT
    • Create a custom assistant
• Contributing

Example

Here is a custom GPT that I quickly made to help answer questions about how to use and integrate Builder.io by simply providing the URL to the Builder docs.

This project crawled the docs and generated the file that I uploaded as the basis for the custom GPT.

Try it out yourself by asking questions about how to integrate Builder.io into a site.

Note that you may need a paid ChatGPT plan to access this feature

Get started

Running locally

Clone the repository

Be sure you have Node.js >= 16 installed.

git clone <https://github.com/builderio/gpt-crawler>

Install dependencies

npm i

Configure the crawler

Open config.ts and edit the url and selectors properties to match your needs.