download web page content (html file) without opening a browser

[Megapro]

Hello, can I download web page content (html file) without opening a browser?

[Megapro]

I’m creating a program that analyzes web page and takes some of it’s content by query

I need to download info from a page without distracting a user

If downloading isn’t possible, maybe you can suggest another solution

[Piotr]

Why do you need browser at all? Just use regular HTTP request. Also are you sure Unity is right tech stack?

[Megapro]

Your asset gives correct page, containing all info

• This reply was modified 4 years, 2 months ago by Megapro.
• This reply was modified 4 years, 2 months ago by Megapro.

[Megapro]

HTTP request cannot give a page loaded through javascript. For example, if you load a youtube page through Web request from Unity it gives you empty page, which does not contain subscriber count.

[Piotr]

Then use javascript to fetch content of a website and pass it to your C# script. There is sample scene which dhows C#-JS communication. However, I still think you might not be using the best tool for this job. Are you sure you want to it on client side?

[bruevp]

Can you please give an example on how to “use javascript to fetch content of a website and pass it to your C# script” without opening a browser window?
It’s probably very simple, but I cannot figure it out without documentation
For now I only found a way to do it after page loads in a browser like this:

private string javaScriptCode = “UnityInAppBrowser.sendMessageFromJS(document.documentElement.outerHTML)”;

void Start() {
InAppBrowserBridge bridge = FindObjectOfType<InAppBrowserBridge>();
bridge.onBrowserFinishedLoading.AddListener(Execute);
bridge.onJSCallback.AddListener(OnMessageFromJS);
}

private void Execute(string command)
{
InAppBrowser.ExecuteJS(javaScriptCode);
}

void OnMessageFromJS(string jsMessage)
{
var filePath = Application.persistentDataPath + “/savedPage.html”;
File.WriteAllText(filePath, jsMessage);
}

[bruevp]

Also, are you suggesting there is a way to prepare a page on server side for download (with loading all content hidden by javascript)?

[Piotr]

There is no way to open a website without actually opening browser window. This plugin definitely wasn’t designed for such use case. What are you trying to accomplish with this HTML parsing?

[bruevp]

Trying to extract some numerical data from different pages for user

[bruevp]

So, is it really hard to add such functionality?

[Piotr]

Not really hard – but it’s definitely not a use case for this plugin.

[Megapro]

But there are no other plugins with such a case, or I couldn’t find them, so I ask You to add this functionality, thanks.

[Piotr]

I’m sure there’s something in C#/.NET/Mono world: https://stackoverflow.com/questions/16642196/get-html-code-from-website-in-c-sharp

[Author]

Posts