Features

• Simple Flask-inspired design - build a scraper with decorators.
• Uses Playwright API - run your scraper in Chrome, Firefox and Webkit and leverage Playwright's powerful selector engine supporting CSS, XPath, text, regex, etc.
• Data grouping - group related results.
• URL pattern matching - run functions on matched URLs.
• Priority - reorder functions based on priority.
• Setup function - enable setup steps (clicking dialogs or login).
• Navigate function - enable navigation steps to move to other pages.
• Custom storage - option to save data to other formats or database.
• Async support - write async handlers.
• Option to use other parser backends aside from Playwright.
  • BeautifulSoup4 - pip install pydude[bs4]
  • Parsel - pip install pydude[parsel]
  • lxml - pip install pydude[lxml]
  • Selenium - pip install pydude[selenium]
• Option to follow all links indefinitely (Crawler/Spider).
• Events - attach functions to startup, pre-setup, post-setup and shutdown events.
• Option to save data on every page.