-
How can I scrape structured data from sites without standard HTML tags?
CSS and XPath selectors allow for flexible selection even if standard tags aren’t used. I customize selectors for each unique element.
-
7 Replies
-
Scrapy’s XPath expressions are especially helpful for locating non-standard elements by their position in the DOM structure.
-
Regular expressions can sometimes capture patterns within unconventional tags, though it’s less reliable for deeply nested data.
-
If data is loaded via JavaScript, I look for embedded JSON or XML, which often contains structured data more easily parsed.
-
Inspecting CSS classes and id attributes for unique identifiers helps locate data within unusual or proprietary tags.
-
For sites with consistent visual patterns, using pixel-based location recognition with headless browsers helps capture data visually.
-
JSON extraction tools can capture data embedded within scripts, which is common on pages that rely on JavaScript for layout.
-
Parsing out meta tags or schema markup is also effective, as some sites embed structured data in the header instead of HTML tables.
Log in to reply.