Main Area and Open Discussion > General Software Discussion

Any free web/cloud solution to monitor a webpage for changes?

(1/3) > >>

kalos:
Hello!

Do you know any free web/cloud solution to monitor a webpage for changes and send you an email notification?

I have been using followthatpage.com but it gives me errors 426, 401 for some websites especially HTTPS.

Thanks!

Shades:
The tool type you search for is called:  Web Scraper.

There are many tools like that, the search engine of your choice will provide more than enough links. Some of these tools have a notification feature, email support is often included.

These tools are not that easy to use/configure, but once you get your head around these, they are pretty reliable at what they do. Realize though all these things you yourself will need to figure out as only you know on which information change has to trigger the notification. As most tools actually provide manuals and sometimes even step-by-step instructions, that shouldn't be too big of a problem.

publicdomain:
... it gives me errors 426, 401 for some websites especially HTTPS.
-kalos (February 20, 2024, 12:50 PM)
--- End quote ---

426 is "upgrade required" (the destination server refuses to accept the current protocol).
401 is "unauthorized", since it needs authentication.

You may want to try another scraper but since cloud based means using someone else's server resources, they are likely to ask for some $ at some point. There is a list of "cloud based web scrapping solutions" at:

* https://geekflare.com/web-scraping-tools/
You can find some free tiers there :up:

Do notice cloud scraper prices can be way overpriced when compared with hosting your own.

The most efficient way is of course having your own web scraper configured in a server under your control -even if the hosting/server fee itself is free. The important part is having the ability to modify the scraper to account for HTML changes and being flexible in the ways to interact with the destination server as time goes by.


Relevant questions:

What frequency do you need to have? (daily, hourly, other)

When you say "monitor a webpage for changes" do you mean any change on the page or is it a certain portion of the contents within the page?

...You can actually get away with using only the "Last-Modified" header in the first case.

If it has to do with monitoring specific contents, then yes, a regular web scraper is due.


BTW I'm coding a "Webpage to Address book" program right now, so I can be of help/assistance since the first part of parsing data from the web is essentially scraping .

Cheers!
Vic

sphere:
Do you know any free web/cloud solution to monitor a webpage for changes and send you an email notification?
-kalos (February 20, 2024, 12:50 PM)
--- End quote ---

You mentioned a web/cloud service.  There is often a free tier that you can use for less than 10 or 25 websites. There are also some selfhosted options. It used to be possible to self-host something like this using some free tier of a cloud service.

But in a pinch if I had to pay I would support an opensource project that had a reasonable pricing model. 

https://github.com/thp/urlwatch  /  https://thp.io/2008/urlwatch/ URLwatch  (I believe open source)
https://github.com/dgtlmoon/changedetection.io   opensource, docker image but I think there are limits of what you can do with selfhosted option.  Noticed this https://torvald.no/web-change-detection when looking for the link.
https://github.com/huginn/huginn a hackable IFTTT  docker
https://github.com/louislam/uptime-kuma  https://uptime.kuma.pet/   Opensource, there is a dem that has some site query information.

kalos:
Thanks but I really need a third-party cloud service, any idea?

Navigation

[0] Message Index

[#] Next page