Getting detected by cloudflare

[vndhote]

undetected chromedriver worked well till yesterday but now, cloudflare improved and the chromedriver is not bypassing cloudflare. I have attached the screenshot of it. cloudflare is just looping the captcha when selenium is running. When I close it, the website loads.

This is my code snippet

It works fine when the script isn't running.
I also tried changing the binary to chrome rather than brave, but the issue still persists

[zakio0403]

I'm on the same spot, was working fine all this month and suddenly today it doesn't work

[MrKhosrojerdi]

I have the same problem as you

[vndhote]

I'm on the same spot, was working fine all this month and suddenly today it doesn't work

I guess it gonna take some time to improve the logics in undetected chrome driver to be undetectable. Anyone having solution to this kindly send me friend request and dm me on discord
My discord Id : vivek9646

[sh-erfan]

Same issue for me. Crawling 24 hours a day during last month, I got stuck a few hours ago.
IMPORTANT: When I close my bot and browser is still open, bypass is successful. This is strange for me.

[benkrej]

Apparently that's an issue with the cloudflare captchas. I found that it works when dev tools are open. Adding the following line resolved the issue for me.
chrome_options.add_argument("--auto-open-devtools-for-tabs") # automatically open dev tools on every new tab
(This is a not a pretty solution ofc, I had no time to dive deeper yet)

[digitalnomad91]

I tried adding the dev-tools option @benkrej but I'm just getting the word "None":

options.add_argument( '--headless' )
options.add_argument( '--auto-open-devtools-for-tabs' )
chrome = uc.Chrome( options = options )
message = chrome.get( 'http://bscscan.com/contractsVerified' )
#chrome.save_screenshot( 'datadome_undetected_webddriver.png' )

print(message)

[benkrej]

You will have to scrape and structure the data yourself, driver.get returns None. I suggest you to look into the coding examples.

[salvecom333]

Apperently that's an issue with the cloudflare captchas. I found that it works when dev tools are open. Adding the following line resolved the issue for me. chrome_options.add_argument("--auto-open-devtools-for-tabs") # automatically open dev tools on every new tab (This is a not a pretty solution ofc, I had no time to dive deeper yet)

god bless you man!

[digitalnomad91]

It worked, thank you!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

[punkerpunker]

chrome_options.add_argument("--auto-open-devtools-for-tabs") # automatically open dev tools on every new tab

That worked mate, thanks!

Btw anyone knows how to close it afterwards? Site-specific protections detect me because it's opened.

I've tried action chains with Shift + Ctrl + i, but didn't succeed

[jacobprice808]

Same issue for me. Crawling 24 hours a day during last month, I got stuck a few hours ago. IMPORTANT: When I close my bot and browser is still open, bypass is successful. This is strange for me.

Does this not indicate that Cloudflare is perhaps tracking mouse movements now?

[vndhote]

Same issue for me. Crawling 24 hours a day during last month, I got stuck a few hours ago. IMPORTANT: When I close my bot and browser is still open, bypass is successful. This is strange for me.

Does this not indicate that Cloudflare is perhaps tracking mouse movements now?

Um.. like I tried using action chains to make it look more natural. I don't think so it may be mouse movements. It is possible that it's detecting selenium.

[vndhote]

chrome_options.add_argument("--auto-open-devtools-for-tabs") # automatically open dev tools on every new tab

That worked mate, thanks!

Btw anyone knows how to close it afterwards? Site-specific protections detect me because it's opened.

I've tried action chains with Shift + Ctrl + i, but didn't succeed

try to use pyautogui to close dev panel
pip install pyautogui
import pyautogui
pyautogui.hotkey("Ctrl", "Shift", "J") add this to your code

[jacobprice808]

Same issue for me. Crawling 24 hours a day during last month, I got stuck a few hours ago. IMPORTANT: When I close my bot and browser is still open, bypass is successful. This is strange for me.

Yeah, it's somehow able to detect the underlying control of Python. When I open a browser with this code and navigate to a Cloudflare site it is looping. When I open a new window manually, I don't experience this issue. Everything else is controlled manually in both cases: navigating to the website, clicking the checkbox, etc. The only difference is how the window was opened.

This is literally all I am doing to open the window:

options = uc.ChromeOptions()
driver = uc.Chrome(options=options)
time.sleep(60)

[ParhamSobhan]

anyone know how can i make its dock at bottom?(automated)(default setting is right)

[benkrej]

I found out that if you open a new tab with js it passes cloudflare captchas. You can close the old tab and use the new one.

driver.execute_script('''window.open("http://nowsecure.nl","_blank");''') # open page in new tab
time.sleep(5) # wait until page has loaded
driver.switch_to.window(window_name=driver.window_handles[0])   # switch to first tab
driver.close() # close first tab
driver.switch_to.window(window_name=driver.window_handles[0] )  # switch back to new tab
time.sleep(2)
driver.get("https://google.com")
time.sleep(2)
driver.get("https://nowsecure.nl") # this should pass cloudflare captchas now

[vndhote]

I found out that if you open a new tab with js it passes cloudflare captchas. You can close the old tab and use the new one.

driver.execute_script('''window.open("http://nowsecure.nl","_blank");''') # open page in new tab
time.sleep(5) # wait until page has loaded
driver.switch_to.window(window_name=driver.window_handles[0])   # switch to first tab
driver.close() # close first tab
driver.switch_to.window(window_name=driver.window_handles[0] )  # switch back to new tab
time.sleep(2)
driver.get("https://google.com")
time.sleep(2)
driver.get("https://nowsecure.nl") # this should pass cloudflare captchas now

God damnnn that worked !
thanks a lot mate

[mutiinsani]

Apperently that's an issue with the cloudflare captchas. I found that it works when dev tools are open. Adding the following line resolved the issue for me. chrome_options.add_argument("--auto-open-devtools-for-tabs") # automatically open dev tools on every new tab (This is a not a pretty solution ofc, I had no time to dive deeper yet)

I tried your solutions, this and the new one and it works on my local machine. However it cannot seem to work on AWS ec2 or Digitalocean droplet. Do you have any pointers where to look to solve this?

[benkrej]

Apperently that's an issue with the cloudflare captchas. I found that it works when dev tools are open. Adding the following line resolved the issue for me. chrome_options.add_argument("--auto-open-devtools-for-tabs") # automatically open dev tools on every new tab (This is a not a pretty solution ofc, I had no time to dive deeper yet)

I tried your solutions, this and the new one and it works on my local machine. However it cannot seem to work on AWS ec2 or Digitalocean droplet. Do you have any pointers where to look to solve this?

Are you sure you are using the exact same code on your local machine and cloud? Try to match the python version as well as uc and chrome versions!

[mutiinsani]

Apperently that's an issue with the cloudflare captchas. I found that it works when dev tools are open. Adding the following line resolved the issue for me. chrome_options.add_argument("--auto-open-devtools-for-tabs") # automatically open dev tools on every new tab (This is a not a pretty solution ofc, I had no time to dive deeper yet)

I tried your solutions, this and the new one and it works on my local machine. However it cannot seem to work on AWS ec2 or Digitalocean droplet. Do you have any pointers where to look to solve this?

Are you sure you are using the exact same code on your local machine and cloud? Try to match the python version as well as uc and chrome versions!

Yes, I use your code to test! Chrome and UC version are exactly the same the only difference is my local Python is version 3.10.10 while my DO server version is 3.10.6 .

I print driver.title for testing and in my local it returns nowSecure while in DO it returns Just a moment... (which is from cloudflare)

[benkrej]

You might need to modify the first sleep, the cf protected site needs to fully load in the new tab in order for it to work.

[mutiinsani]

Been trying changing that, still no luck

[Sunda001]

Apparently that's an issue with the cloudflare captchas. I found that it works when dev tools are open. Adding the following line resolved the issue for me. chrome_options.add_argument("--auto-open-devtools-for-tabs") # automatically open dev tools on every new tab (This is a not a pretty solution ofc, I had no time to dive deeper yet)

thank u man!!! it works fine for me!!!

[sh-erfan]

Same issue for me. Crawling 24 hours a day during last month, I got stuck a few hours ago. IMPORTANT: When I close my bot and browser is still open, bypass is successful. This is strange for me.

Does this not indicate that Cloudflare is perhaps tracking mouse movements now?

I click the checkbox with mouse, also applied random waits to simulate real human, but still stuck. I guess it is detecting selenium somehow

[sh-erfan]

Same issue for me. Crawling 24 hours a day during last month, I got stuck a few hours ago. IMPORTANT: When I close my bot and browser is still open, bypass is successful. This is strange for me.

Yeah, it's somehow able to detect the underlying control of Python. When I open a browser with this code and navigate to a Cloudflare site it is looping. When I open a new window manually, I don't experience this issue. Everything else is controlled manually in both cases: navigating to the website, clicking the checkbox, etc. The only difference is how the window was opened.

This is literally all I am doing to open the window:

options = uc.ChromeOptions()
driver = uc.Chrome(options=options)
time.sleep(60)

I guess that is strictly related to detecting type of the handler that controls the browser, may be via JS. When I close bot, then the handler is kind of disposed, and so, CF realizes no automatic handler is back there and let me bypass. Can you test this scenario: Open the main window via code, then open a new tab and head to the same website manually.

[benkrej]

Same issue for me. Crawling 24 hours a day during last month, I got stuck a few hours ago. IMPORTANT: When I close my bot and browser is still open, bypass is successful. This is strange for me.

Yeah, it's somehow able to detect the underlying control of Python. When I open a browser with this code and navigate to a Cloudflare site it is looping. When I open a new window manually, I don't experience this issue. Everything else is controlled manually in both cases: navigating to the website, clicking the checkbox, etc. The only difference is how the window was opened.

This is literally all I am doing to open the window:

options = uc.ChromeOptions()

driver = uc.Chrome(options=options)

time.sleep(60)

I guess that is strictly related to detecting type of the handler that controls the browser, may be via JS. When I close bot, then the handler is kind of disposed, and so, CF realizes no automatic handler is back there and let me bypass. Can you test this scenario: Open the main window via code, then open a new tab and head to the same website manually.

My second workaround does exactly what you described.

[benkrej]

Good to know that it works with playwright!

[Sunda001]

@benkrej Hi, For now, I can use playwright to pass the cf challenge without opening the debugging tool.

About cf turnstile captcha:

• In the first case, after detecting the environment, you can pass without clicking.
• In the second case, you need to click to confirm the real person before you can pass. (This case is related to the proxy ip you use, and elite proxies generally do not appear).
• The last case requires you to identify specific things in the picture. In this case, I haven't encountered it yet while using playwright.

note that your way need to use headless=False, so it means it will fail when someone uses headless mode.

[NCLnclNCL]

I found out that if you open a new tab with js it passes cloudflare captchas. You can close the old tab and use the new one.

driver.execute_script('''window.open("http://nowsecure.nl","_blank");''') # open page in new tab
time.sleep(5) # wait until page has loaded
driver.switch_to.window(window_name=driver.window_handles[0])   # switch to first tab
driver.close() # close first tab
driver.switch_to.window(window_name=driver.window_handles[0] )  # switch back to new tab
time.sleep(2)
driver.get("https://google.com")
time.sleep(2)
driver.get("https://nowsecure.nl") # this should pass cloudflare captchas now

God damnnn that worked ! thanks a lot mate

not working bro

[benkrej]

@NCLnclNCL try the dev tools workaround then

[digitalnomad91]

Both workarounds are no longer working, fyi everyone. I'm sure it's being worked on, but in the meantime I just created a small browser extension that opens up the page I want and then sends the HTML to a little proxy server which writes the results to a file that my other processes can read. I knew that this was going to happen, especially with "workarounds" like the two that were found, but I had hoped for a bit more time. CF team is really going in on this "war of attrition". It reminds me a lot back in like 2015-2016 when I played a similar game with Google engineers and with streaming video from Google Drive accounts. Unfortunately it ended up being a losing game for me in the long term, but I think with AI being where it is currently that ultimately CF will lose this war. If anyone is interested I can make a repo with my browser extension and proxy server with some instructions on how to set it up as a temporary workaround. The downside is that you'll need a computer @ home (or wherever I suppose) that can have a browser up and running as often as you need to get the latest contents of that page. I still have to do some testing to see how it does over time, particularly when I'm afk, but I think worst case I might just have to add some mouse jiggle or something like that. If I start getting captcha'ed then obviously that will only work if I'm around, but otherwise the page I need generally loads automatically after a couple redirects from CF. If I end up running into the captcha problem I might setup some kind of push notification / chat server bot thing that can notify me and then setup a way to be able to "solve" the captcha from wherever I'm at. Combine that with a few other people besides myself running the same thing and hopefully we'll have a "workaround" that can last a lot longer, even if it's only to get us through periods like this where we're "losing".

[ilike2burnthing]

I'm not sure the digital communications agency based in Glasgow, Scotland needed to be tagged in this. 😉

[digitalnomad91]

| I'm not sure the digital communications agency based in Glasgow, Scotland needed to be tagged in this.

lmao, my bad tag removed. Still not fully awake yet it seems, thanks for the quick heads up. You're one of the people working on a more permanent solution I believe, right? If so then thanks a lot, I know it's definitely appreciated by a lot of people as well.

[NCLnclNCL]

Both workarounds are no longer working, fyi everyone. I'm sure it's being worked on, but in the meantime I just created a small browser extension that opens up the page I want and then sends the HTML to a little proxy server which writes the results to a file that my other processes can read. I knew that this was going to happen, especially with "workarounds" like the two that were found, but I had hoped for a bit more time. CF team is really going in on this "war of attrition". It reminds me a lot back in like 2015-2016 when I played a similar game with Google engineers and with streaming video from Google Drive accounts. Unfortunately it ended up being a losing game for me in the long term, but I think with AI being where it is currently that ultimately CF will lose this war. If anyone is interested I can make a repo with my browser extension and proxy server with some instructions on how to set it up as a temporary workaround. The downside is that you'll need a computer @ home (or wherever I suppose) that can have a browser up and running as often as you need to get the latest contents of that page. I still have to do some testing to see how it does over time, particularly when I'm afk, but I think worst case I might just have to add some mouse jiggle or something like that. If I start getting captcha'ed then obviously that will only work if I'm around, but otherwise the page I need generally loads automatically after a couple redirects from CF. If I end up running into the captcha problem I might setup some kind of push notification / chat server bot thing that can notify me and then setup a way to be able to "solve" the captcha from wherever I'm at. Combine that with a few other people besides myself running the same thing and hopefully we'll have a "workaround" that can last a lot longer, even if it's only to get us through periods like this where we're "losing".

Yes bro, i need extention

[vvanglro]

Passing the challenge with debug turned on seems to have been fixed by cf.

There are already cf people who are paying attention to related discussions and project, so if we continue to openly pass the challenge method, it will undoubtedly be a tug of war.😪

I have announced that stop maintaining my project. vvanglro/cf-clearance#58
Although I tried a new way to pass the challenge. I can't make it public because it will be fixed by cf.
My idea is to provide an api service that can happily pass the challenge without making it public.

[baptx]

I found out that if you open a new tab with js it passes cloudflare captchas. You can close the old tab and use the new one.

driver.execute_script('''window.open("http://nowsecure.nl","_blank");''') # open page in new tab
time.sleep(5) # wait until page has loaded
driver.switch_to.window(window_name=driver.window_handles[0])   # switch to first tab
driver.close() # close first tab
driver.switch_to.window(window_name=driver.window_handles[0] )  # switch back to new tab
time.sleep(2)
driver.get("https://google.com")
time.sleep(2)
driver.get("https://nowsecure.nl") # this should pass cloudflare captchas now

I just needed to do driver.execute_script('window.open("http://nowsecure.nl/#relax");') (the #relax is not important for the issue but for another issue #934) and then by a doing a sleep of around 30 seconds, I had time to complete the CAPTCHA manually.
By using a custom profile, the CAPTCHA will not be asked for some time after we completed it (I did not check for how long yet). However it does not work on all websites using Cloudflare CAPTCHA: #1455.
Does someone have an idea why we need to open a new tab to avoid detection? I wonder which JavaScript code Cloudflare is using to detect this because in a normal web browser, we don't need "window.open".

Update: if we don't use a custom profile, switching tabs is still necessary if we want to execute JavaScript code like an alert (https://www.geeksforgeeks.org/execute_script-driver-method-selenium-python/) on the target website but opening another domain name like in the solution from benkrej was not necessary. The first tab switch was not necessary either because WebDriver has control on it by default even if it is showing the second tab.

[NCLnclNCL]

I found out that if you open a new tab with js it passes cloudflare captchas. You can close the old tab and use the new one.

driver.execute_script('''window.open("http://nowsecure.nl","_blank");''') # open page in new tab
time.sleep(5) # wait until page has loaded
driver.switch_to.window(window_name=driver.window_handles[0])   # switch to first tab
driver.close() # close first tab
driver.switch_to.window(window_name=driver.window_handles[0] )  # switch back to new tab
time.sleep(2)
driver.get("https://google.com")
time.sleep(2)
driver.get("https://nowsecure.nl") # this should pass cloudflare captchas now

I just needed to do driver.execute_script('window.open("http://nowsecure.nl/#relax");') (the #relax is not important for the issue but for another issue #934) and then by a doing a sleep of around 30 seconds, I had time to complete the CAPTCHA manually. By using a custom profile, the CAPTCHA will not be asked for some time after we completed it (I did not check for how long yet). Does someone have an idea why we need to open a new tab to avoid detection? I wonder which JavaScript code Cloudflare is using to detect this because in a normal web browser, we don't need "window.open".

because the driver is in another tab, if you switch to the tab you just opened you will be detected

[baptx]

@NCLnclNCL Do you know which JavaScript code they can use to detect this?

[benkrej]

@ultrafunkamsterdam is working on it as it seems so it may be worth to wait for his release.
#1420

[baptx]

@benkrej The workaround with the tabs switch is still needed with the new version 3.5.2. I also have another issue: #1455.

[benkrej]

You have to find a workaround yourself everything published is getting patched very soon.

[baptx]

@benkrej Maybe there are things they cannot patch. If I find a workaround, I think I will share it.

[benkrej]

Maybe it would be good to approach it another way and try to fix uc so it won't get detected in the first place without any temporary solutions. Would be awesome to have a chromedriver again like it was for the longest time!

[baptx]

@benkrej I found a workaround: #1455 (comment)
I hope I did not make a mistake by sharing it :D

[benkrej]

@baptx Works only with visible challenges some sites have no visible challenge and therefore are still stuck in the infinite loop. Glad you found a way that works for you!

[miha1llumi]

I found out that if you open a new tab with js it passes cloudflare captchas. You can close the old tab and use the new one.

driver.execute_script('''window.open("http://nowsecure.nl","_blank");''') # open page in new tab
time.sleep(5) # wait until page has loaded
driver.switch_to.window(window_name=driver.window_handles[0])   # switch to first tab
driver.close() # close first tab
driver.switch_to.window(window_name=driver.window_handles[0] )  # switch back to new tab
time.sleep(2)
driver.get("https://google.com")
time.sleep(2)
driver.get("https://nowsecure.nl") # this should pass cloudflare captchas now

thanks, bro. Everything works.

[iprahka]

I found out that if you open a new tab with js it passes cloudflare captchas. You can close the old tab and use the new one.

driver.execute_script('''window.open("http://nowsecure.nl","_blank");''') # open page in new tab
time.sleep(5) # wait until page has loaded
driver.switch_to.window(window_name=driver.window_handles[0])   # switch to first tab
driver.close() # close first tab
driver.switch_to.window(window_name=driver.window_handles[0] )  # switch back to new tab
time.sleep(2)
driver.get("https://google.com")
time.sleep(2)
driver.get("https://nowsecure.nl") # this should pass cloudflare captchas now

Youre absolutely right, but at the the same time I just tried to comment out the '--headless' line and it works perfect too without any long loading

Here's my working code version:

from selenium import webdriver
import time
from config import proxy_ip, proxy_port # you can remove it and the option below

options = webdriver.ChromeOptions()
# options.add_argument("--headless")
options.add_argument(f"--proxy-server={proxy_ip}:{proxy_port}")
options.add_argument('--user-data-dir=/home/iprahka/.config/google-chrome/Default')

driver = undetected_chromedriver.Chrome(options)

driver.get('https://www.nowsecure.nl')


time.sleep(99)

[Ahmed-Ashraf-Khalil]

i saw this tool online garantee cloudflare verfication to be solved

cons:
the documentation is in chinese and couldn't link it with selenium.

[n0nameNPC]

Apperently that's an issue with the cloudflare captchas. I found that it works when dev tools are open. Adding the following line resolved the issue for me. chrome_options.add_argument("--auto-open-devtools-for-tabs") # automatically open dev tools on every new tab (This is a not a pretty solution ofc, I had no time to dive deeper yet)

I tried your solutions, this and the new one and it works on my local machine. However it cannot seem to work on AWS ec2 or Digitalocean droplet. Do you have any pointers where to look to solve this?

are you using package for rendering ? For example: Xvfb

[Ghxst]

Apperently that's an issue with the cloudflare captchas. I found that it works when dev tools are open. Adding the following line resolved the issue for me. chrome_options.add_argument("--auto-open-devtools-for-tabs") # automatically open dev tools on every new tab (This is a not a pretty solution ofc, I had no time to dive deeper yet)

I tried your solutions, this and the new one and it works on my local machine. However it cannot seem to work on AWS ec2 or Digitalocean droplet. Do you have any pointers where to look to solve this?

are you using package for rendering ? For example: Xvfb

I am, I can relate to having issues automating my browsers whenever the system is headless and I rely on Xfvb.
Any ideas why or tips to avoid detection of headless systems?