Grabbing files from authenticated website

[kalos]

Hello

I want to grab xls files via specific links through a website that has a form authentication. Because it has an additional sms authentication step, it will not be possible to fully automate a live connection.

The session is set at 60 minutes so I am thinking if I ping/query the website every 30min, the cookie will renew/refresh.

However, this pinging will need to come from a 24/7 server and not my laptop as I cannot have it on all the time.

Is there any free solution for this?

I was thinking to create a Sharepoint Flow where it will download the xls every 30 min refreshing the cookie/auth/sessionID each time. I was hoping that I can somehow login manually for the first time via Sharepoint and then since the flow will be scheduled every 30, it will run forever refreshing the session.

Or do you have any other idea?
I was checking some of the free Azure services if they are suitable but no luck.

Thanks!

[publicdomain]

... that has a form authentication.

-kalos (September 23, 2021, 05:18 AM)

Hello Kalos! What's the URL/Link to the authentication form? Chances are it can be automated via a browser automation script + a GUI.

Cheers!

[kalos]

Hi!

Unfortunately it's a proprietary site and I cannot tell it.

It is a form authentication website which has an additional step of authentication so it is really really hard to automate it.

Instead, I am just looking to download specific urls every 30 min through the IE/Edge so that the session does not expire.

What would be the best way to do that? I tried Powershell, Windows Task Scheduler, Excel Data Connections but non really worked due to downloading either nothing or the login page.

[publicdomain]

Unfortunately it's a proprietary site and I cannot tell it.

-kalos (September 23, 2021, 03:02 PM)

No problem

What would be the best way to do that?

-kalos (September 23, 2021, 03:02 PM)

I would either use Selenium (https://www.selenium.dev) or a simple "embedded" WebView/WebBrowser for this.

If you're in a rush, we can discuss privately, no biggie. As long as the resulting program goes to serve the public community, I'm in! 

(BTW, no hard-coded private URLs, just configurable fields)

[Shades]

Hello

I want to grab xls files via specific links through a website that has a form authentication. Because it has an additional sms authentication step, it will not be possible to fully automate a live connection.

The session is set at 60 minutes so I am thinking if I ping/query the website every 30min, the cookie will renew/refresh.

However, this pinging will need to come from a 24/7 server and not my laptop as I cannot have it on all the time.

Is there any free solution for this?

I was thinking to create a Sharepoint Flow where it will download the xls every 30 min refreshing the cookie/auth/sessionID each time. I was hoping that I can somehow login manually for the first time via Sharepoint and then since the flow will be scheduled every 30, it will run forever refreshing the session.

Or do you have any other idea?
I was checking some of the free Azure services if they are suitable but no luck.

Thanks!

-kalos (September 23, 2021, 05:18 AM)

Although it isn't a safe practice, you can add your credentials directly into the URL of the site you wish to visit.

Like so:
https://<user name>:<password>@<website URL>

or in a example:
https://kalos:[email protected]

If needed, you can even add a different port number:   https://kalos:[email protected]:8080