Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Add support for undetectable browsers #512

Open
plord12 opened this issue Nov 30, 2024 · 1 comment
Open

[Feature]: Add support for undetectable browsers #512

plord12 opened this issue Nov 30, 2024 · 1 comment

Comments

@plord12
Copy link

plord12 commented Nov 30, 2024

Webscraping has become quite tricky recently, with CDN's detecting bots pretty easily. It would be great if we could add support for one of the "undetectable" browsers. For example -

Does the upstream have similar features?

Camoufox does support playwright, but only with python.

Is your feature request related to a problem? Please describe.

Many websites now spot playwright-go / playwright-go-stealth and block access.

Describe the solution you'd like

In an ideal world, it would be as simple as -

playwright.Install(&playwright.RunOptions{Browsers: []string{"camoufox"}})
...
pw.Camoufox.Launch()

Additional context

Camoufax v132.0-beta.15 and before supplied a launch binary which was usable with playwright-go by simply setting ExecutablePath :

pw.Firefox.Launch(playwright.BrowserTypeLaunchOptions{ExecutablePath: playwright.String("launch"))})

However launch has since been removed.

To keep my scraping going, I've taken the launch source into my project, https://github.com/plord12/webscrapers (specifically https://github.com/plord12/webscrapers/tree/main/launch and https://github.com/plord12/webscrapers/blob/main/utils/utils.go#L140). This works with the latest Camoufox release.

@dlouvier
Copy link

dlouvier commented Dec 8, 2024

I'm new user of playwright-go but I was able to run it with camoufox as server, but you need to run it over two different processes:

Install the latest version of camoufox

pip install -U camoufox[geoip]==0.4.6

Then install the latest (at this moment, not yet published) version of playwright-go

go get github.com/playwright-community/playwright-go@b6313db

And install the drivers (on my case, this fails because I use Fedora, however it still installs the driver)

go run github.com/playwright-community/playwright-go/cmd/playwright@b6313db install --with-deps

Then you need to start camoufox as server

Launching server...
Server launched: 683.392ms
Websocket endpoint: ws://localhost:38835/de22fd1ffd383c1d3669beada9e7eb73 

Configure the Endpoint in your script:

package main

import (
	"log"
	"github.com/playwright-community/playwright-go"
)

func main() {
	pw, err := playwright.Run()
	if err != nil {
		log.Fatalf("oh lala: %v", err)
	}

	browser, err := pw.Firefox.Connect("ws://localhost:38835/de22fd1ffd383c1d3669beada9e7eb73")
	if err != nil {
		log.Fatalf("could not launch browser: %v", err)
	}

	page, err := browser.NewPage()
	if err != nil {
		log.Fatalf("could not create page: %v", err)
	}
	if _, err = page.Goto("http://127.0.0.1:8080"); err != nil {
		log.Fatalf("could not goto: %v", err)
	}

	if _, err = page.; err != nil {
		log.Fatalf("could not goto: %v", err)
	}
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants