Any free web/cloud solution to monitor a webpage for changes?

[kalos]

Hello!

Do you know any free web/cloud solution to monitor a webpage for changes and send you an email notification?

I have been using followthatpage.com but it gives me errors 426, 401 for some websites especially HTTPS.

Thanks!

[Shades]

The tool type you search for is called:  Web Scraper.

There are many tools like that, the search engine of your choice will provide more than enough links. Some of these tools have a notification feature, email support is often included.

These tools are not that easy to use/configure, but once you get your head around these, they are pretty reliable at what they do. Realize though all these things you yourself will need to figure out as only you know on which information change has to trigger the notification. As most tools actually provide manuals and sometimes even step-by-step instructions, that shouldn't be too big of a problem.

[publicdomain]

... it gives me errors 426, 401 for some websites especially HTTPS.

-kalos (February 20, 2024, 12:50 PM)

426 is "upgrade required" (the destination server refuses to accept the current protocol).
401 is "unauthorized", since it needs authentication.

You may want to try another scraper but since cloud based means using someone else's server resources, they are likely to ask for some $ at some point. There is a list of "cloud based web scrapping solutions" at:

• https://geekflare.com/web-scraping-tools/

You can find some free tiers there

Do notice cloud scraper prices can be way overpriced when compared with hosting your own.

The most efficient way is of course having your own web scraper configured in a server under your control -even if the hosting/server fee itself is free. The important part is having the ability to modify the scraper to account for HTML changes and being flexible in the ways to interact with the destination server as time goes by.


Relevant questions:

What frequency do you need to have? (daily, hourly, other)

When you say "monitor a webpage for changes" do you mean any change on the page or is it a certain portion of the contents within the page?

...You can actually get away with using only the "Last-Modified" header in the first case.

If it has to do with monitoring specific contents, then yes, a regular web scraper is due.


BTW I'm coding a "Webpage to Address book" program right now, so I can be of help/assistance since the first part of parsing data from the web is essentially scraping .

Cheers!
Vic

[sphere]

Do you know any free web/cloud solution to monitor a webpage for changes and send you an email notification?

-kalos (February 20, 2024, 12:50 PM)

You mentioned a web/cloud service.  There is often a free tier that you can use for less than 10 or 25 websites. There are also some selfhosted options. It used to be possible to self-host something like this using some free tier of a cloud service.

But in a pinch if I had to pay I would support an opensource project that had a reasonable pricing model. 

https://github.com/thp/urlwatch  /  https://thp.io/2008/urlwatch/ URLwatch  (I believe open source)
https://github.com/d...n/changedetection.io   opensource, docker image but I think there are limits of what you can do with selfhosted option.  Noticed this https://torvald.no/web-change-detection when looking for the link.
https://github.com/huginn/huginn a hackable IFTTT  docker
https://github.com/louislam/uptime-kuma  https://uptime.kuma.pet/   Opensource, there is a dem that has some site query information.

[kalos]

Thanks but I really need a third-party cloud service, any idea?

[publicdomain]

Thanks but I really need a third-party cloud service, any idea?

-kalos (February 28, 2024, 01:31 PM)

Idea: self-host with a Linux VPS in the cloud.

[dantheman]

Have never tried it as registered user, but it does have basic free option and seems to be able to check even sites without RSS.

https://newsblur.com/

[4wd]

You can also run your own: changedetection.io

Web Site Change Detection, Restock monitoring and notifications.

Detect website content changes and perform meaningful actions - trigger notifications via Discord, Email, Slack, Telegram, API calls and many more.

[kalos]

I found this which seems interesting: https://webalert.me/

It says it will work as if you visit the website yourself with your browser so it will work with cookies, web forms etc.

I have not tried it yet, will see!

[Shades]

Do you know any free web/cloud solution to monitor a webpage for changes and send you an email notification?

-kalos (February 20, 2024, 12:50 PM)

You mentioned a web/cloud service.  There is often a free tier that you can use for less than 10 or 25 websites. There are also some selfhosted options. It used to be possible to self-host something like this using some free tier of a cloud service.

But in a pinch if I had to pay I would support an opensource project that had a reasonable pricing model. 

https://github.com/thp/urlwatch  /  https://thp.io/2008/urlwatch/ URLwatch  (I believe open source)
https://github.com/d...n/changedetection.io   opensource, docker image but I think there are limits of what you can do with selfhosted option.  Noticed this https://torvald.no/web-change-detection when looking for the link.
https://github.com/huginn/huginn a hackable IFTTT  docker
https://github.com/louislam/uptime-kuma  https://uptime.kuma.pet/   Opensource, there is a dem that has some site query information.

-sphere (February 21, 2024, 08:28 AM)

• Huginn - Not easy to use. I have the Docker deployed in a Proxmox environment. Haven't spent all too much time with it, but the examples I saw when glancing over those, they looked involved.
• UpTime Kuma - Run that also in the same Docker setup. This is not really useful for tracking content. It is great in hooking up any kind of service you run inside your network to let it check if it is up or not. You can then use webhooks to get informed by UpTime-Kuma via its interface, mail, Teams/Slack/Discord, Telegram and so many other (web-)services. In that guise, UpTime-Kuma is excellent. But for tracking changes in web-based content, not so much.

[Deozaan]

You can also run your own: changedetection.io

Web Site Change Detection, Restock monitoring and notifications.

Detect website content changes and perform meaningful actions - trigger notifications via Discord, Email, Slack, Telegram, API calls and many more.

-4wd (March 23, 2024, 06:28 AM)

Combine the above with this:

https://www.oracle.com/cloud/free/