Use online webdriver for BeautifulSoup / Selenium Python code (vs. local reference)

by

I wrote local Python code in Spyder that does webscraping using BeautifulSoup and Selenium. I now want to transfer that code to run online and on a schedule (using pythonanywhere). That works fine for the pure BeautifulSoup elements. The parts that use Selenium, currently have some configuration on my local setup, including a local reference to the webdriver, see below.

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
driver = webdriver.Chrome("C:/Users/my.name/Downloads/chromedriver-win64/chromedriver-win64/chromedriver.exe", options=options)
driver.get('URL TO SCRAPE')

When I transfer the code online, I obviously can’t refer to my local C drive anymore. Is there such a thing as an online version of the webdriver exe file I can reference?

When I asked ChatGPT, I was referred to services like BrowserStack and SauceLabs, but (without reading their website too much) this looks a little overkill when seeing the price points.

Any advice or pointers are appreciated – thanks!

  • What version of Selenium are you using? If recent, you shouldnt need to specify the driver location

    – 

  • thanks, locally i have 3.141. online (at pythonanywhere), it’s 4.1.5. when i run the command without the file path (but still with options), i get the following: File “<stdin>”, line 1, in <module> File “/usr/local/lib/python3.10/site- […] (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

    – 




  • also tried Firefox and Ie configurations: – FF: “Message: ‘geckodriver’ executable needs to be in PATH. ” – Ie: ‘IEDriverServer.exe’ executable needs to be in PATH. Please download from selenium.dev/downloads and read up at github.com/SeleniumH Q/selenium/wiki/InternetExplorerDriver

    – 

tl;dr: If you are using Selenium 4.6.0 or greater, you do not need to specify the driver location; if you are using an older version, upgrading could be a quick fix.

With 4.6.0 Selenium introduced Selenium Manager, which handles setting up browser drivers, removing the necessity to do it yourself.

From the documentation:

Setting up a browser driver once is not that complicated, but as
browser release cycles got shorter, and now we have a new
Chrome/Firefox/Edge version every 4-6 weeks, the task of keeping the
browser driver in sync with the browser version is not that easy
anymore.

Selenium Manager is a new tool that helps to get a working environment
to run Selenium out of the box. Beta 1 of Selenium Manager will
configure the browser drivers for Chrome, Firefox, and Edge if they
are not present on the PATH.

So, while you can ensure your pythonanywhere environment is properly set up in their console and get file paths from there, you could also simply upgrade your selenium to 4.6.0.

Leave a Comment