LangChain Web Scraping

LangChain is a noted framework for building applications powered by language models. It has become a solid solution for a variety of tasks. With LangChain web scraping, supported by other tools like BeuaitfulSoup and Puppeteer, you can create intelligent web scraping pipelines.

You also can contextualize and analyze raw data more efficiently, allowing you to achieve more with the data you have collected. In this guide, we’ll discuss how LangChain can become a pivotal tool in the way you use web scraping.

What LangChain Can Do for You 

LangChain has the ability to manage memory and interactions with ease. This makes it helpful for navigating dynamic websites in web scraping. It also helps web scrapers overcome complex multi-step processes like filling out forms or moving through login pages to get to the content you desire and want to access. 

LangChain web scraping provides a number of benefits to users. Specifically, it helps those engaged in more robust web scraping setups achieve more with less. By combining LangChain with proxy solutions that we offer at Rayobyte, it is possible to enhance access to target websites while also reducing the risk of bans. 

The Perfect Partner for LangChain Scraping

View Our Proxies

When you do this (which we will discuss in more detail in just a bit), you create an opportunity to reduce the risk of bans, scale your web scraping efforts, improve efficiency, and capture more robust data you can use to meet your specific needs. 

Why Use LangChain for Web Scraping

LangChain scraping has its benefits and numerous applications. We encourage you to use web scraping only for ethical strategies and research without violating a website’s terms of service. The following are some of the best advantages of using LangChain web scraping to achieve your specific goals:

Setup is easy: Though you will need some experience and insight into web scraping, LangChain itself is a user-friendly interface that does not require a significant learning curve. It helps to reduce many of the technical challenges that can sometimes make web scraping difficult for those who may not have a long history of coding experience. 

You do not need a lot of experience to use LangChain. It is intuitive enough that you can apply your basic skills to achieve the scraping project design and functionality you need. 

Natural language queries: Another benefit to using LangChain for web scraping is that it enables you to use natural language queries that can provide instruction to the web scraper. In many situations, this is a critical advantage over traditional strategies for several reasons. To do this otherwise, you need to write complex code and then do so over and over again.

With LangChain, you can use natural language queries – such as describing what you want – and the tool will generate the necessary code for you. It’s much like using AI for prompts for other applications you may be using now. That makes it far easier for those who may not have a coding background.

Versatile use: LangChain is very versatile in its overall applications and functionality. It can be used for a wide range of tasks, though most people tend to focus only on specific sites or data types.

However, sometimes, the information you need is located on other sites, such as social media or an e-commerce website. You can use LangChain scraping to make scraping news feeds easier, as well. With LangChain scraping, there’s more opportunity to capture the diverse information and resources you need.

Scaling is easy: As your success with web scraping grows, you’ll likely want to use it to achieve more of your goals. You can use this strategy for even the largest projects you want to navigate. 

On the flip side, LangChain is still very effective for smaller projects. You can put your scraping tool in place quickly and navigate through your project with ease. Then, scale it as you need to collect more data or navigate more complex strategies. 

All The Proxies You Need

Rotating, static, residential or data center? Pair your LangChain scraping with our versatile proxies.

Take A Look

Overall, LangChain for web scraping provides you with better functionality and more robust operations. It can certainly streamline the process of scraping content from a website but also help you capture that content from multiple websites at the same time. Instead of having to build new code for each of the sites you need to scrape, you can use the tool’s abilities to provide specific goals. For example, LangChain can simplify a task by letting you define a schema that can then be universally applied. This allows you to navigate more of the sites you need without needing constant updates.

If you are new to web scraping, LangChain is a good starting point. You do not have to have any previous knowledge or skill of how to build a web scraper or when to use these tools to use this tool. That’s because of the super effective and user-friendly design. The natural language processing features also make this a very effective strategy. 

Keep in mind that there are some key strategies you will need to employ to get the most out of the process. For example, LangChain’s document loader uses Playwright to work. This can lead to issues when websites notice what you are doing and begin to block you. This can delay scraping actions. The good news is there are ways to get around this. One of the simplest routes is to use the requests library for scraping, which eliminates the strain that can happen from Playwright.

Web Scraping with LangChain: How to Get Started

Now that you can see how web scraping with LangChain can work for you, the question is, how do you get started? To provide you with some insight, we will use a few demonstrations below. You can use LangChain to scrape content from a website. To show you what to do, we are going to use example.com as a website – be sure you update this code with your own. For our project, we are going to scrape data that is found on a category page within that fake website. 

Download what you need to use LangChain first. You will need to download the following:

!pip install -q openai langchain playwright beautifulsoup4 tiktoken

!playwright install

!playwright install-deps

It is not uncommon to encounter some async issues during this initial process. If you do, then you can use the following to get past them:

import nest_asyncio

nest_asyncio.apply()

Now, to engage with this process, we are going to use OpenAI’s GPT-3.5 turbo 0613, or in short, the Open AI Language Model. To do this, you will need to set up your API key. Use the following code to get you started:

import os

from langchain.chat_models import ChatOpenAI

os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"

llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")

It also becomes necessary to use a few key tools to help with LangChain scraping. That includes AsyncChromiumLoader, which will allow for the asynchronous loading of web pages. With it in place, you can then fetch web content asynchronously. 

You also need BeautifulSoupTransformer. This tool will help you parse the data and transform the HTML code that you receive. The BeautifulSoup library is an excellent choice for parsing HTML and XML documentation. 

So, to get these tools, you will need to use the following code:

from langchain.document_loaders import AsyncChromiumLoader

from langchain.document_transformers import BeautifulSoupTransformer

loader = AsyncChromiumLoader(["https://www.example.com/search?q=tablet"])

docs = loader.load()

bs_transformer = BeautifulSoupTransformer()

docs_transformed = bs_transformer.transform_documents(

        docs, tags_to_extract=["div","span","h2","a"]

    )

Now that you have this started, there are a few key things to keep in mind. If you look at this code, you will notice that a single URL is used as a parameter. In this case, we are using example.com and looking for “tablet” on it. 

The loader object will load the HTML content from the URL at the same time and store it in the HTML variable, which contains the raw HTML content for the example.com website search results page. 

This bit of code will also transform the HTML document to the format we prefer: BeautifulSoup. That’s using the BeautifulSoup Transformer function, as noted in the code above. That means that it does not pull all the HTML content from the side. Instead, it looks at the specific HTML tags, including, in this case, span, div, and others. When it finds these tags on the example.com website, it captures that information. 

To tell the code to do this, we include these in the tags_to_extract parameter. That helps to reduce the overall workload of the process and improves efficiency.

Dealing with Large Amounts of Content

One of the limitations you may run into relates to the need to pass large texts of content to OpenAI. This does not always offer ideal results. So, we navigate around this, and you can use a few other key tools available to help you. That includes RecursiveCharacterTextSplitter. It sounds more complex than it is. What it does is create a way to divide the data into more management chunks of information. That makes it easier to interact with that content. It also may help you get around restrictions that are often in place to block such transactions that include huge amounts of data. 

Here’s what the code looks like for this:

splitter=RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=1000, chunk_overlap=0)

splits = splitter.split_documents(docs_transformed)

The chunk_size parameter allows us to instruct the tool on how much information we are splitting. In this case, we’re talking about 1000, which means that just 1000 tokens, maximum, will be contained. There’s another key feature here – chunk_overlap. When we set this to 0, we are telling the code that there is no reason for any adjacent chunks to overlap.

Creating Your Data Extraction Data Schema

What you will find is that there’s no complexity here related to CSS selectors or XPath. You do not have to search for this on your own. Instead, you can outline the schema by specifying the appropriate keywords and the data types you want to extract. 

Here’s an example:

from langchain.chains import create_extraction_chain

schema = {

    "properties": {

        "product_title": {"type": "string"},

        "product_mrp": {"type": "integer"},

        "product_description":{"type": "array", "items": {"type": "string"}},

        "product_reviews_count":{"type": "string"}

    },

    "required": ["product_title","product_mrp","product_description"],

}

def extract(content: str, schema: dict):

    return create_extraction_chain(schema=schema, llm=llm).run(content)

You can now extract the content using the defined schema. This is the extract function that you can use to do this:

extracted_content = extract(schema=schema, content=splits[0].page_content)

Now that you have all of that in place, it’s rather easy to see how it may work for your project. When you use Langchain web scraping, you can pull these pieces together efficiently. 

It will pull the content that you specify and allow you to use that data for anything you need.

When to Use LangChain for Web Scraping

If you are unsure when and how to use web scraping with LangChain, reach out to our team. You can use it for so many different AI-drive workflows, including:

• Integration of LLMs with huge data sources
• Analyze data with it
• Summarization
• Answer questions 

A Note About Proxies

As we have discussed many times, when it comes to web scraping with LangChain or otherwise, you need to protect yourself and your tasks using proxies. A proxy service works to protect your IP address. It makes it hard for any other website to see how it is requesting this information and pulling data from the site.

Ready To Scrape?

Ethically-sourced proxies ideal for your next LangChain web scraping project.

Take A Look

Proxies from Rayobyte can help with web scraping in many ways. We encourage you to learn more about incorporating proxies into the web scraping process to add protection and create more of an authentic outcome. If you are looking for help with LangChain web scraping or you are ready to jump on board, learn more about how Rayobyte can help you.You can use all of our web scraping tutorials to get started building your information tool and optimize results with ease. Contact us now to learn more about Rayobyte proxies.